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Preface 


My  interest  in  air  combat  simulator  training  began  as  a  practical  problem  while 
serving  as  the  Fighter  Weapons  Phase  Head  for  the  F/A-18  Fleet  Readiness  Squadron 
(Fighter  Training  Unit  equivalent)  on  a  three-year  Navy  exchange  tour  at  Lemoore  NAS, 
California.  While  overhauling  their  air-to-air  syllabus,  I  was  allocated  a  limited  number 
of  syllabus-hours  to  train  students  in  networked  F/A-18  simulators.  The  simulators  were 
enclosed  in  spherically  shaped  rooms,  providing  a  realistic  cockpit  field-of-view.  I  was 
pelted  with  suggestions  from  other  instructors  on  how  the  simulators  should  be  used  in 
the  new  training  plan.  Unfortunately,  opinion  was  the  only  data  source  for  decision 
making — there  was  no  documented  guidance  on  how  to  improve  air  combat  training 
through  simulation. 

This  document  is  presented  to  serve  as  a  knowledge  base  for  those  endeavoring  to 
develop  F-15C  syllabuses  for  Distributed  Mission  Training  (DMT).  Armed  with  lessons 
learned  from  years  of  simulator  study,  F-15C  wings  can  develop  effective  and  credible 
training  programs.  Blending  the  best  features  of  both  DMT  and  flight  training  into 
effective  training  programs  will  be  key  for  swaying  pilots  who  currently  shun  expansion 
of  simulator  training. 

I  owe  a  great  deal  of  thanks  for  the  insights  of  Dr.  Herbert  Bell  and  Dr.  Ronald 
Dunlap  at  the  Air  Force  Research  Laboratory,  and  Lt  Col  “Taco”  Bell  at  the  33  Fighter 
Wing  at  Eglin  AFB.  I  also  want  to  thank  Maj  “Slap”  Maxwell  at  HQ  ACC/DOTO,  and 
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many  folks  at  the  29th  Training  Systems  Squadron — specifically  Lt  Col  Mike  Graham, 
Mr.  Jeff  Wakefield,  Mr.  Rick  Griffin,  and  Rick  “Slick”  Eplett — for  their  assistance.  I 
must  also  thank  Col  Chenoweth  for  his  guidance  in  this  project,  helping  me  narrow  my 
focus  into  something  useful  for  the  Combat  Air  Forces  (CAF).  Finally,  my  family 
deserves  the  greatest  thanks,  for  their  endless  patience  and  encouragement. 
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Abstract 

Air  Combat  Command  is  investing  in  Distributed  Mission  Training  (DMT)  to 
provide  realistic  mission  training  to  the  Combat  Air  Forces  (CAF)  using  advances  in 
simulation  technology.  DMT  will  network  advanced  simulators  (and  some  real-world 
systems)  to  provide  combat  aircrews  with  team  training  in  a  synthetic  wartime 
environment.  F-15C  units  will  be  first  in  the  CAF  to  incorporate  DMT;  they  are 
confronted  with  developing  training  programs  utilizing  this  new  tool  without  previous 
experience  of  how  to  exploit  the  benefits  of  simulation  for  air  combat  training.  This 
paper  seeks  to  assist  syllabus  developers  by  providing  a  summary  of  lessons  learned  from 
years  of  air  combat  simulation  study,  and  applying  those  lessons  to  DMT. 

A  comprehensive  analysis  of  air  combat  simulation  training  studies  provides  insight 
on  how  to  improve  F-15C  air  combat  training  using  DMT.  Studies  not  only  demonstrate 
simulators  can  provide  effective  training,  but  also  identify  unique  benefits  and  limitations 
of  simulator  training,  and  offer  several  training  program  considerations  to  achieve  the 
most  effective  results.  DMT  benefits  not  available  in  flight  training  include  unique 
instruction  options,  the  capability  to  repeatedly  practice  desired  tasks  or  missions,  and  the 
opportunity  to  train  in  complex  combat  scenarios.  Identified  limitations  include  sources 
of  negative  training  and  the  lack  of  physiological  stresses  or  inputs  that  may  be  desired  in 
training.  For  training  programs,  simulator  studies  suggest  which  tasks  should  be 
emphasized  in  DMT,  how  metrics  can  identify  training  needs,  who  is  expected  to  gain  the 
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most  benefit  from  DMT,  and  how  to  intermix  DMT  and  flight  scheduling  to  optimize 
training. 
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Chapter  1 


Introduction 

I  am  convinced  Modeling  and  simulation  Technologies  available  today 
will  enable  us  to  significantly  change  the  way  we  train  in  the  future....  We 
need  to  take  a  hard  look  at  how  this  technology  will  change  our  training 
philosophy  . . . 

— General  Michael  E.  Ryan 

The  DMT  Vision 

Air  Combat  Command  (ACC),  while  making  tremendous  improvements  in  aircraft 
and  weapons  capabilities,  is  beginning  to  outgrow  its  contemporary  means  of  training. 
Flight  training  is  no  longer  sufficient  for  preparing  combat  crews  for  the  complexities  of 
modern  warfare.  “Safety  considerations,  mission  complexity,  airspace  and  range 
restrictions,  real-world  commitments,  and  cost  limit  the  effectiveness  of  live  flying 
training  opportunities.”1  To  help  compensate  for  these  limitations,  ACC  is  investing  in 
Distributed  Mission  Training  (DMT)  to  provide  realistic  mission  training  to  the  Combat 
Air  Forces  (CAF)  using  advances  in  simulation  technology." 

DMT  will  network  advanced  simulators  (and  some  real-world  systems)  to  provide 
combat  aircrews  with  team  training  in  a  synthetic  wartime  environment,  no  longer 
restricting  training  to  single-ship  operations.  Mission  Training  Centers  (MTC)  will  house 
multiple  simulators  for  each  weapon  system,  to  be  used  for  individual  and  team  training. 
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DMT  will  also  combine  MTCs  in  a  wide-area  network,  allowing  inter-team  training  such 
as  composite-force  and  AEF  missions.  The  wide-area  network  will  be  managed  by  a 
Distributed  Warfighting  Center  (DWC)  allowing  distributed  events  (between  MTCs)  and 
stand-alone  events  (within  a  single  MTC)  to  be  conducted  concurrently.  During  all 
training  events,  additional  forces  (both  adversary  and  friendly)  can  be  represented  by 
human-controlled  interactive  threat  stations,  or  by  computer-controlled  constructive 
simulations.  Eventually,  personnel  using  their  real-world  operational  systems  (such  as 
command  and  control  systems)  will  also  be  integrated  as  live  participants  in  DMT. 
(MTC  capabilities  and  the  DMT  expansion  plan  are  elaborated  in  Appendix  A  and  B, 
respectively.) 


A  New  Training  Tool 

It  is  possible  that  a  presumed  utility  of  simulators  may  stifle  innovations  to  make 
best  use  of  DMT.  In  the  past,  simulators  were  used  primarily  for  single-ship  training,  “to 
learn  some  basics  about  the  weapon  system,  learn  how  to  start  the  motor,  how  to  employ 
the  radar,  learn  which  button  does  what;  but  [not  to]  learn  the  essence  of  [the  fighter] 
business,  which  is  team  combat.”4  DMT  finally  provides  the  CAF  with  the  capability  for 
team  training,  providing  seemingly  endless  possibilities  for  air  combat  training.  Because 
DMT  will  seem  so  realistic,  a  natural  tendency  may  be  to  treat  MTC  simulators  just  like 
real  aircraft.  But  simulators  are  not  the  same  as  aircraft — simulators  may  be  limited  in 
some  respects,  yet  may  have  some  capabilities  beyond  those  of  flight  training  that  can  be 
exploited  to  improve  training. 

MTC  accreditation  will  identify  some  of  these  differences  by  evaluating  how 
accurately  the  system  simulates  real-world  interactions,  such  as  “the  quality  of  the  visual 
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cues,  accuracy  of  the  sensor  presentations,  environmental  factors,  validity  of  threat 
models,  fidelity  of  the  cockpit,  and  the  overall  hardware/software  capabilities.”5  ACC 
will  then  use  these  evaluation  results  to  decide  how  DMT  can  contribute  to  annual 
training  and  proficiency  requirements.6  Accreditation  will  therefore  determine  some 
MTC  limitations  and  allowable  use  for  accomplishing  Ready  Aircrew  Program  (RAP) 
sorties  and  events,  but  will  not  dictate  how  combat  units  will  use  DMT  on  a  daily  basis 
for  its  continuation  training  (CT).  Operational  units  are  responsible  for  developing  their 
own  training  profiles  and  programs  for  DMT,  ‘‘based  on  expected  wartime  tasking, 
employment  concepts,  and  MAJCOM  training  policy.”7  They  are  faced  with  the 
challenge  of  utilizing  this  new  training  tool  without  previous  experience  of  how  to  exploit 
DMT  benefits  or  avoid  its  limitations. 

Although  operational  units  have  never  had  access  to  networked  simulators  for  daily 
training,  simulators  have  been  used  over  the  last  20  years  for  specialized  air  combat 
training.  In  fact,  many  studies  have  been  conducted  to  measure  the  effectiveness  of  this 
simulator  training.  The  purpose  of  this  paper  is  to  propose  how  F-15C  air  combat 
training  can  be  improved  by  applying  lessons  learned  from  previous  simulator  training 
studies  to  DMT. 

A  comprehensive  review  of  documented  air-to-air  simulator  training  research  will  be 
presented  to  uncover  lessons  applicable  to  future  training  plans  incorporating  DMT.  A 
description  of  the  simulator  facilities,  participants,  procedures,  and  results  will  be 
presented  for  each  study.  In  many  cases  interesting  secondary  findings  or  study 
discussions  provide  additional  insight  for  improving  air  combat  simulator  training.  The 
studies  not  only  demonstrate  that  simulators  can  provide  effective  training,  but  also 
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identify  unique  benefits  and  limitations  of  simulator  training,  who  achieves  the  most 
benefit  from  such  training,  what  tasks  are  best  accomplished  in  simulation,  and  how 
simulator  and  flight  training  should  be  intermixed  to  maximize  learning. 

This  paper  will  address  the  implications  of  these  findings  for  DMT.  DMT  may  be 
laden  with  the  flaws  and  limitations  of  past  and  current  simulators.  This  investigation 
will  attempt  to  identify  such  flaws  by  comparing  DMT  simulator  hardware  specifications 
to  those  recommended  in  previous  studies.  Also,  due  to  the  lack  of  experience  by 
operational  units  with  network  capable  simulators,  it  is  plausible  that  many  assumptions 
about  the  utility  of  the  new  training  devices  will  lead  to  less  than  optimum  training 
strategies,  reducing  the  potential  effectiveness  of  DMT.  This  investigation  will  search  for 
training  strategies  suggested  in  air  combat  training  effectiveness  studies  that  can  be 
incorporated  into  F-15C  training  plans  for  DMT. 

Once  units  determine  how  best  to  exploit  DMT,  they  will  be  better  able  to  determine 
an  effective  balance  of  aircraft  and  simulator  sorties  to  prepare  pilots  for  combat. 
Although  this  report  specifically  addresses  air  combat  training  for  F-15C  units,  findings 
are  likely  to  have  implications  for  nearly  all  CAF  fighters,  providing  them  a  foundation 
for  developing  effective  training  plans  as  they  are  integrated  into  the  CAF-wide  DMT 
network. 

Notes 

1  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  Distributed  Mission  Training,”  13  October  1998,  19. 

‘  Gen  Richard  E.  Hawley,  commander,  Air  Combat  Command,  address  to  the 
National  Training  Systems  Association  19th  Interservice/Industry  Training  Simulation 
and  Education  Conference,  Orlando,  Fla.,  2  December  1997. 

3  HQ  ACC/DOTO,  “Concept  of  Operations  for  Distributed  Mission  Training.” 

4  Hawley. 
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Notes 


5  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  F-15C  Mission  Training  Centers,”  Working  Draft,  29 
September  1998,  25. 

6  Ibid.,  22. 

7  Ibid.,  20. 
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Chapter  2 


Review  of  Simulator  Training  Studies 

Simulator  studies  can  be  divided  into  three  categories:  utility  evaluations,  in¬ 
simulator  learning  studies,  and  transfer  of  training  studies.1  Utility  evaluations  are  the 
most  basic  of  the  three,  conducted  by  survey  to  analyze  pilot  opinions  of  simulator 
training.  In-simulator  learning  (ISL)  studies  demonstrate  learning  achieved  within  a 
simulator  environment  by  measuring  improvements  in  performance,  while  transfer  of 
training  (TOT)  studies  demonstrate  improved  flight  performance  due  to  simulator 
training. 


Utility  Evaluations 

Early  ACES  Surveys  in  the  SAAC 

The  earliest  utility  evaluations  involved  nothing  more  than  simple  aircrew  surveys 
conducted  after  training  in  the  Simulator  for  Air-to-Air  Combat  (SAAC).  The  SAAC 
combined  two  F-4  cockpits  for  training  one-versus-one  basic  fighter  maneuvers  (BFM), 
or  two-versus-one  air  combat  maneuvers  (ACM)  against  a  constructive  (computer- 
controlled)  threat.  Cockpit  visual  displays  provided  296-degree  horizontal  and  150- 
degree  vertical  field  of  view.' 
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Tactical  Air  Command  (TAC)  conducted  one-week  Air  Combat  Engagement 
Simulation  (ACES)  courses  for  mission-ready  (MR)  pilots  using  the  SAAC  in  the  late 
1970’ s.  Several  surveys  indicated  ACES  BFM  and  ACM  training  was  effective.  “The 
overwhelming  consensus  from  the  participants  was  that  the  training  was  quite  valuable.”4 
Although  these  early  surveys  were  encouraging,  they  did  not  provide  insight  to  improve 
training;  later  studies  were  conducted  looking  specifically  for  ways  to  improve  air  combat 
simulator  training. 

MACAIR  Utility  Evaluations 

From  1988  to  1990,  Armstrong  Laboratory  (now  Air  Force  Resources  Laboratory, 
AFRL)  conducted  two  detailed  surveys  at  what  was  the  McDonnell  Aircraft  (MACAIR) 
simulation  facility  in  St.  Louis,  Missouri.  This  facility  combined  a  weapons  director 
(WD)  station  with  two  F-15  cockpits  suspended  in  40-foot  domes  providing  pilots  “with 
a  nearly  full  field  of  view.”5  Less  sophisticated  simulator  cockpits  were  available  for 
controlling  interactive  threats  against  the  F-15  pilots.  Constructive  threats  were 
combined  with  interactive  threats  in  each  scenario,  challenging  F-15  pilots  with  a  large 
number  of  adversaries.  The  facilities  allowed  pilots  to  conduct  multi- ship  training 
against  multiple  air  and  surface-to-air  missile  (SAM)  threats  in  a  variety  of  unrestricted 
air  combat  missions.6 

The  two  studies  involved  a  total  of  136  MR  F-15A/C  pilots.  After  one  week  of 
MACAIR  simulator  training,  pilots  were  asked  to  rate  the  value  of  both  their  unit 
continuation  training  (CT)  and  the  F-15  simulator  training  for  a  number  of  combat  tasks. 
Ratings  were  based  on  a  five-point  scale  from  1 -’’Unacceptable”  to  5-”Excellent.”  The 
results  not  only  indicated  the  training  was  worth  while,  but  also  identified  many  combat 
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tasks  perceived  to  be  better  trained  in  the  simulators  than  in  CT.  These  tasks  also 


received  a  CT  value  rating  of  less  than  3-”Adequate.”  The  tasks  found  better  suited  for 
simulator  training  are  listed  in  table  1.  Simulator  training  was  also  perceived  as  highly 

o 

valuable  for  other  tasks,  but  not  necessarily  more  valuable  than  CT. 


Table  1.  MAC  AIR  Simulator  Training  Ratings  of  Combat  Tasks 


Better  Trained  in  the  Simulator 

Highly  Valuable  Simulator  Training 

Multibogey,  Four  or  More 

Reaction  to  SAMs 

Dissimilar  Air  Combat  Tactics 

All-Weather  Employment 

ECM/ECCM  Employment 

Communications  Jamming 

Low  Altitude  Tactics 

Chaff/Flare  Employment 

Escort  Tactics 

Tactical  Electronic  Warfare  System  Assessment 
Work  with  WD 

All-Aspect  Defense 

BVR  Employment 

Radar  Sorting 

Reaction  to  Air  Interceptors 

Missile  Employment 

Egress  Tactics 

Source:  Adapted  from  Michael  R.  Houck,  Gary  S.  Thomas,  and  Herbert  H.  Bell, 
Training  Evaluation  of  the  F-15  Advanced  Air  Combat  Simulation  (Williams  Air  Force 
Base,  AZ:  Armstrong  Laboratory,  Aircrew  Training  Research  Division,  September 
1991),  table  6,10. 

Overall,  the  simulator  training  value  was  rated  “quite  uniformly  across  pilot 
background  variables  with  the  exception  of  a  few  tasks  which  were  rated  significantly 
higher  by  low-time  pilots:  Visual  Identification,  Electronic  Countermeasures/Electronic 
Counter-Countermeasures  (ECM/ECCM)  Employment,  and  Reaction  to  SAMs.  In 
summary,  the  rating  data  indicate  much  agreement  among  CONUS  F-15  pilots  regarding 
their  perception  of  their  training  needs.”9 

Pilots  also  rated  what  they  thought  the  benefit  would  be  for  pilots  of  other  experience 
levels,  using  a  scale  from  l-”Not  Beneficial”  to  5-”Extremely  Beneficial.”  The  simulator 
training  was  rated  “Highly  Beneficial”  (4.2  or  better)  for  all  experience  levels  with  the 
exception  of  new  wingman — rated  only  “Very  Beneficial”  (3.8  mean  rating).  Further 
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analysis  determined  that  four-ship  leads  were  the  primary  source  of  lower  ratings  for  new 
wingmen.  “Four-ship  leads  rated  the  benefit  of  . . .  training  lower  for  new  wingmen  than 
did  the  wingman  themselves,”  commenting  that  the  MACAIR  training  “may  be  too 
demanding  for  a  new  wingman.”10 

Combining  tasks  into  a  realistic,  complex  scenario  enhanced  training. 

“Overwhelmingly,  pilots  thought  the  mission  scenarios  were  the  key  ingredient  to  the 

training  value  ...  being  challenging,  demanding,  realistic,  and  providing  appropriate 

numbers  of  air  threats  for  multibogey  training.”11  The  ability  to  practice  these  scenarios 

numerous  times  were  also  found  to  be  a  unique  benefit  of  the  simulator  training.  “Pilots 

benefited  greatly  from  repeated  exposures  to  the  same  scenarios,  and  considered  the 

1 2 

variety  of  scenarios  available  in  the  simulator  to  be  a  major  advantage.” 

“An  especially  important  advantage  of  simulator-based  training  was  the  availability 
of  immediate  performance  feedback  that  is  not  available  in  airborne  training.”  Real¬ 
time  kill  removal  allowed  pilots  to  verify  properly  flown  tactics,  and  see  the 
consequences  of  poorly  executed  tactics  or  weapons  employment. 

In  addition  to  pilot-identified  benefits  of  the  simulator  training,  the  study  also  listed 
simulation  features  perceived  to  be  unrealistic  or  potential  sources  of  negative  training 
(producing  undesirable  habits  or  behaviors14).  The  list  primarily  identified  hardware  and 
software  deficiencies  specific  to  the  MACAIR  facility: 15 

1.  Red/white  color  changes  of  projected  aircraft  images  were  confusing  and 
projections  allowed  tallies  at  unrealistic  distances 

2.  F-15A/C  pilots  had  to  adapt  to  a  modified  F-15E  simulator  cockpit 

3.  Visually  accommodating  and  resolving  projected  targets  was  difficult 

4.  Information  provided  by  WDs  was  too  precise  to  be  realistic 

5.  Manned  threats  were  not  challenging  or  credible 

6.  Computer-controlled  threats  were  too  proficient 
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Despite  these  shortcomings,  the  simulator  training  overall  was  considered  to  be  a 
tremendously  valuable  training  tool,  providing  much  needed  training  not  available  by 
other  means.  While  these  utility  studies  provide  some  insight  into  the  value  of  air-to-air 
simulator  training,  they  do  not  address  whether  a  pilot’s  performance  actually  improved 
due  to  that  training.  In-simulator  learning  studies  accomplish  this  by  measuring 
performance  improvements  taking  place  “as  a  function  of  practice  within  the 
simulation.”16 

In-Simulator  Learning  (ISL)  Studies 
1981  SAAC  ISL  Study 

The  earliest  ISL  study  was  conducted  in  1981  by  the  USAF  Tactical  Fighter 
Weapons  Center  at  Nellis  AFB,  Nevada.  This  study  used  the  SAAC  to  demonstrate 
performance  improvements  in  the  simulator  flying  BFM  and  ACM  against  a  constructive 
threat.  The  study  “reported  significant  improvements  in  weapons  employment  ... 
[including]  quicker  first  shots  ...  more  valid  shots,  and  fewer  missed  shot 
opportunities.”17 

F-14  ACMS  ISL  Study 

The  next  year  a  Navy  study  demonstrated  ISL  in  the  F-14  Air  Combat  Maneuvering 
Simulator  (ACMS)  at  Oceana  NAS,  Virginia.  The  ACMS  combined  two  40-foot  domes 
similar  to  the  MACAIR  simulators,  but  could  only  project  a  single  threat  image, 
restricting  it  to  BFM  or  ACM.  The  threat  could  be  controlled  by  computer,  console- 
operator,  or  from  the  other  dome  simulator. 
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The  study  measured  aircrew  performance  using  an  All-Aspect  Maneuvering  Index 
(AAMI)  proving  an  overall  score  “based  upon  a  formula  that  incorporates  range,  antenna- 
train-angle,  and  angle-off-the-tail  as  the  major  variables”  A  higher  AAMI  score 
indicated  a  more  offensive  position  at  the  conclusion  of  each  engagement.  The  test 
involved  experienced  F-14  aircrew  flying  BFM  against  a  computer-controlled  threat. 
Three  engagements  were  used  as  a  pre-test  prior  to  receiving  40  training  engagements. 
An  identical  test  was  given  following  the  training,  resulting  in  scores  “much  higher  than 
pre-test  scores,  [indicating]  that  the  training  ...  in  the  simulator  does,  in  fact,  improve 
performance.”19 

1990  SAAC  ISL  Study 

In  1990  a  second  SAAC  ISL  study  “also  demonstrated  significant  improvements  in 
performance  as  a  function  of  simulator  training  ...  using  the  AAMI  as  the  primary 
measure  of  performance.”20 

MACAIR  ISL  Study 

The  MACAIR  facility  was  also  used  to  conduct  ISL  studies.  The  multiple-threat 
capability  of  this  facility  allowed  Armstrong  Laboratory  to  study  offensive  and  defensive 
counter-air  (OCA  &  DCA)  missions.  F-15  pilots  were  paired  into  16  elements  for  three 
days  of  “intense  simulation  training.”'  Controlled  scenarios  were  flown  before  and  after 
the  training,  revealing  “significantly  higher”  performance  after  training.'  Measures  used 
for  the  comparison  included  F-15  survival,  mission  success  (e.g.  percentage  of  adversary 
bombers  destroyed  prior  to  reaching  their  target  in  DCA  missions),  and  mission 
efficiency  (weighted  exchange  ratios),  as  depicted  in  figure  1. 


11 


1 00% 


80% 

60% 

40% 

20% 

0% 


n  Pretest 
n  Posttest 


Survival 


Success 


Efficiency 


Figure  1.  DCA  Pre/Post  Test  Performance  Comparison 
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1993  MULTIRAD  ISL  Study 

Armstrong  Laboratory  conducted  another  ISL  study  starting  in  1993  using  their  own 
multiship  simulation  facility  (MULTIRAD)  at  Williams  AFB  (now  Williams  Gateway 
Airport)  in  Mesa,  Arizona.  This  facility  combined  two  high  fidelity  F-15  cockpits  (360- 
degree  visual)  with  two  F-16  (adversary)  cockpits  and  a  WD  station.'  Numerous 
constructive  friendly  and  threat  aircraft  were  added  to  the  scenarios.  As  part  of  a  larger 
program  researching  situational  awareness  (SA),  this  study  examined,  among  other 
things,  “the  potential  of  multiship  simulation  as  a  tool  for  training  SA.”'  To  measure 
performance,  an  SA  rating  scale  (SARS)  was  developed  to  identify  24  “behavioral 
indicators”  of  a  pilot’s  SA  (table  2).  SA  was  defined  as  “a  pilot’s 
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Table  2.  Categories  and  Indicators  used  in  SARS 


Category 

Behavioral  Indicators 

Tactical  Game  Plan 

Developing  plan 

Executing  plan 

Adjusting  plan  on-the-fly 

System  Operation 

Radar 

Tactical  electronic  warfare  system 
Overall  weapons  system  proficiency 

Communication 

Quality  (brevity,  accuracy,  timeliness) 
Ability  to  use  controller  information 

Information  Interpretation 

Interpreting  vertical  situation  display 
Interpreting  threat  warning  system 
Ability  to  use  controller  information 
Integrating  overall  information 

Radar  sorting 

Analyzing  engagement  geometry 

Threat  prioritization 

Tactical  Employment-BVR 

Targeting  decisions 

Fire-point  selection 

Tactical  Employment-Visual 

Maintain  track  of  bogeys/friendlies 
Threat  evaluation 

Weapons  employment 

Tactical  Employment-General 

Assessing  offensiveness/defensiveness 
Lookout 

Defensive  reaction 

Mutual  support 

Source:  Adapted  from  Wayne  L.  Waag  et  al,  “Use  of  Multiship 
Simulation  as  a  Tool  for  Measuring  and  Training  Situation 
Awareness,”  in  AGARD-CP-575,  Situation  Awareness:  Limitations 
and  Enhancement  in  the  Aviation  Environment,  (Neuilly-Sur-Seine, 

France:  Advisory  Group  for  Aerospace  Research  and  Development, 

January  1996),  table  1,  20-2. 

continuous  perception  of  self  and  aircraft  in  relation  to  the  dynamic  environment  of 
flight,  threats,  and  mission,  and  the  ability  to  forecast,  then  execute  tasks  based  on  that 
perception.”26 

Forty  MR  F-15  flight  leads  (study  subjects)  and  23  wingmen  flew  36  engagements, 
progressing  incrementally  in  complexity.  Subject  matter  experts  (SME)  observed  each 
engagement  and  rated  subject  pilots  on  each  of  the  24  behavioral  indicators.  Each 
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Indicator  was  graded  on  a  six-point  scale  from  1 -’’Acceptable”  to  6-”Outstanding.”27  An 
SA  score  was  then  derived  using  SA  Rating  Scales  (SARS).28  A  comparison  of  two 
missions  flown  the  first  and  last  day  indicated  a  statistically  significant  improvement  in 
2v2  DC  A  missions,  but  only  a  small  (not  statistically  significant)  improvement  in  2v4 
OCA  missions  (figure  2). 


□  First  Engagement 
□Last  Engagement 


Figure  2.  Effects  of  Practice  on  Observer  SA  Ratings29 

Because  each  mission  increased  in  complexity,  the  trend  was  actually  a  decrease  in 
SA  scores  as  the  study  progressed.  However,  after  weighting  the  scores  for  all  scenarios 
according  to  difficulty  (determined  by  the  SMEs),  the  seven  simulator  sorties  were  shown 
to  progressively  improve  SA  (figure  3). 

Armstrong  Laboratory  surveyed  the  pilots  for  information  similar  to  the  1990  utility 
study.  All  pilots  rated  the  training  to  be  beneficial,  but  training  was  rated  more  beneficial 
for  pilots  upgrading  to  a  higher  qualification  level,  such  as  a  wingman  in  Mission 
Qualification  Training 
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Simulator  Mission/Scenario 

Figure  3.  SA  Scores  Weighted  for  Scenario  Difficulty  Across  Missions30 

(MQT)  or  upgrading  to  two-ship  flight  lead,  or  a  two-ship  flight  lead  upgrading  to  four- 
ship  lead  (figure  4). 


Figure  4.  Rated  Benefit  of  Training  for  Various  Levels  of  Experience31 

ISL  studies  to  date  suggest  learning  is  taking  place  in  air-to-air  simulator  training, 
and  is  further  supported  by  survey  data,  but  success  within  a  given  simulator  environment 
does  not  necessarily  mean  learning  in  the  simulator  will  improve  task  performance  in 
flight.  Simulator  learning  must  also  improve  performance  in  the  aircraft. 

Transfer  of  Training  (TOT)  Studies 

Transfer  of  training  (TOT)  studies  measure  performance  improvements  in  flight  due 
to  simulator  training.  Sometimes  referred  to  as  “transfer  of  learning,”  these  studies 
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show  simulator  training  (or  learning)  influencing  later  behavior  in  an  aircraft.  Ideally  this 
influence  will  reinforce  desired  behavior,  but  it  can  also  create  undesired  habits  or 
behavior — negative  training  transfer.34 

Control  Groups 

TOT  studies  generally  involve  two  groups:  an  experimental  group  and  a  control 
group.  The  experimental  group  receives  simulator  pre-training  before  being  tested  or 
trained  further  in  an  aircraft,  while  the  control  group  begins  aircraft  testing  or  training 
without  simulator  influence.  The  resulting  performance  difference  between  the  two 
groups  indicates  whether  the  simulator  influence  was  positive  or  negative  for  that 
training. 

The  control  groups  do  not  replace  missing  simulator  training  with  flight  training;  in 
fact,  the  control  group  is  expected  not  to  receive  any  additional  flight  training  during  the 
time  the  experimental  group  is  accomplishing  simulator  sorties.  If  the  control  group  is 
given  any  flight  training,  the  same  flight  training  should  also  be  given  to  the  experimental 
group  to  ensure  the  only  variable  between  the  groups  is  simulator  training. 

The  nature  of  TOT  studies  is  to  demonstrate  that  tasks  learned  or  improved  in  a 
simulator  can  be  applied  to  actual  flight.  TOT  studies  do  not  compare  simulator  training 
to  flight  training ;  they  make  no  attempt  to  prove  that  simulation  is  better  than  in-flight 
training.  However,  one  implication  of  positive  transfer  is  that  “use  of  the  simulator  can 
reduce  dependence  upon  operational  aircraft  during  training  by  influencing  the  learning 
of  tasks  that  must  be  performed  in  those  aircraft.”  In  other  words,  if  simulator  training 
results  in  more  efficient  flight  training,  fewer  flights  should  be  required  to  achieve  the 
same  objectives  (or,  the  same  number  of  flights  can  be  used  to  reach  higher  objectives). 
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1975  Navy  TOT  Study 

The  first  simulator- to-flight  TOT  study  on  air  combat  training  was  conducted  in 
1975  with  Navy  F-4J  students.  This  was  the  first  simulator  to  use  a  remotely  controlled 
adversary.  The  experimental  group  practiced  eight  BFM  maneuvers  against  a  single 
adversary  during  six  simulator  sorties.  Instructors  graded  pilot  performance  of  each 
maneuver  using  12  “performance  variable”  (PV)  criteria  on  a  scale  of  zero-”F”  to  12- 
”A+.”  Instructors  also  indicated  one  of  five  possible  final  positions  obtained  at  the  end  of 
each  engagement:  firing  position,  lag  pursuit  position,  offensive,  neutral,  or  defensive.37 

After  training  the  experimental  group  on  all  simulator  events,  both  experimental  and 
control  groups  flew  six  BFM  sorties  for  aircraft  training  in  the  same  maneuvers — again 
graded  on  PVs  and  final  positions  for  each  maneuver.  Overall,  “the  experimental  group 
achieved  a  more  advantageous  average  position,  came  to  an  AIM-9  final  position  more 
often,  and  required  fewer  maneuvers  to  achieve  firing  positions  than  did  the  control 
group.”  In-flight  grades  for  the  experimental  group  averaged  higher  for  each  of  the 
sorties  flown  as  well  as  for  each  maneuver,  although  the  differences  were  slight  (figure 
5). 

Although  the  experimental  group  averaged  better  grades  in  all  maneuvers,  only  the 
rolling  scissors  maneuver  demonstrated  a  statistically  reliable  difference.  One  postulated 
cause  for  the  slight  differences  in  the  remaining  maneuvers  was  a  lack  of  control  during 
in-flight  sorties,  such  as  variations  in  the  number,  order,  and  types  of  maneuvers  flown 
during  each  syllabus  mission.  It  was  also  postulated  that  transfer  results  would  have  been 
higher  if  not  “diluted”  by  a  long  time  delay  between  simulator  training  and  subsequent 
flight  training.  Third,  it  was  believed  that 
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sequencing  each  flight  to  follow  its  corresponding  simulator  event  would  have  yielded 
better  results. 


Figure  5.  BFM  Maneuver  Grades39 

Although  the  rolling  scissors  maneuver  was  considered  to  be  most  difficult,  it 
achieved  the  highest  transfer — a  38%  improvement  in  grades  between  experimental  and 
control  groups.  In  addition,  rolling  scissors  maneuvers  were  most  hampered  in 
simulation  by  a  limited  visual  display,  making  its  high  transfer  even  more  remarkable. 
Although  an  isolated  demonstration,  “food  for  thought”  was  offered:  “transfer  effects 
may  be  augmented  when  the  pretraining  task  (simulator  pretraining)  is  somewhat  more 
difficult  than  the  ultimate  task  (aircraft  retraining).”40  It  was  also  suggested  that 
simulator  fidelity  is  less  important  than  how  the  simulator  is  utilized.  “Transfer  of 
learning  seems  likely  more  a  function  of  goodness  of  instruction  than  a  fidelity  of 
simulation  . . .  [involving]  use  of  the  powerful  technique  of  training  for  transfer:  training 
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‘around’  (or  in  spite  of)  limitations  that  are  . . .  inherent  in  the  simulator.”  (Emphasis  in 
original)41 

Individual  performance  variables  were  also  analyzed  to  determine  if  there  were 
significant  grade  differences  between  groups.  Taking  PV  grades  for  all  maneuvers  into 
consideration,  only  one  variable,  “nose  position  at  pull  down,”  demonstrated  a 
statistically  reliable  difference;  none-the-less,  all  twelve  PVs  were  rated  better  for  the 

42 

simulator-trained  group  (figure  6).  “This  consistent  superiority  can  not  be  ignored.” 


Figure  6.  BFM  Performance  Variable  Grades43 

By  isolating  PV  grades  for  each  BFM  maneuver,  two  PVs  were  found  responsible 
for  distinguishing  the  rolling  scissors  maneuver  above  the  rest.  These  were  two  PVs 
“instructors  considered  most  important:  Use/Control  of  Energy  and  Nose  to  Tail 
Separation.  The  superiority  of  the  experimental  group  in  the  rolling  scissors  was,  then, 
achieved  by  enhancing  performance  on  important  variables.”44 
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The  study  also  hoped  to  determine  the  number  of  aircraft  training  hours  that  could  be 
replaced  by  an  hour  of  simulator  training  by  “proficiency  advancing”  or  skipping  some 
aircraft  time  for  experimental  group  pilots,  but  was  unable  to  do  so  because  “the  study 
was  simply  too  short.”45 

1977  SAAC  TOT  Study 

Air  Force  Human  Resources  Laboratory  (AFHRL)  utilized  SAAC  F-4  simulators  for 
a  TOT  study  in  1977,  conducted  primarily  as  a  platform  motion  study.  Student  pilots 
were  placed  into  three  groups:  six  were  given  no  simulator  training,  eight  were  given 
simulator  training  with  motion,  and  eight  more  were  given  simulator  training  without 
motion.  IPs  graded  each  student’s  performance  of  nine  “skills”  and  ten  BFM  “tasks”  on  a 
scale  of  zero  to  nine.  (Skills  and  tasks  were  similar  to  the  PVs  and  maneuvers  used  in  the 
1975  Navy  TOT  study.) 46 

Some  in-simulator  learning  was  demonstrated  (although  not  statistically  reliable)  by 
comparing  performance  measures  of  the  first  and  last  simulator  sorties.  AFHRL  was 
unable  to  measure  in-flight  learning  due  to  a  limited  number  of  available  sorties — each 
sortie  increased  in  difficulty,  with  no  spare  sorties  to  repeat  the  first  profile  for  a  pre/post 
test  comparison.  Of  primary  interest,  comparing  in-flight  performance  between  the 
control  group  and  the  two  simulator-trained  groups  failed  to  demonstrate  an  effective 
transfer  of  training:  “The  data  indicated  that  the  SAAC  training  did  not  seem  to  increase 
instructor  ratings  of  performance  in  the  aircraft.”47 

AFHRL  considered  the  small  transfer  effects  to  be  a  trend  in  BFM  simulator  TOT 
studies,  meaning  they  were  not  very  successful  at  finding  variations  in  performance.  It 
referenced  the  1975  Navy  TOT  study  (above)  and  an  additional  study  conducted  during 
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SAAC  FOT&E  student  training  (without  reference  or  dates),  where  “a  comparison  of 
instructor  ratings  from  BFM  training  yielded  ...  a  very  small  positive  effect.”  A  lack  of 
objective  measures  was  also  offered  as  explanation  for  the  failure  to  demonstrate 
significant  transfer.  Task  and  skill  scores  were  based  purely  on  the  subjective  judgement 
of  the  same  IP  giving  instruction  from  the  adversary  simulator  cockpit;  instructor  ratings 
may  not  have  been  “sufficiently  sensitive  [or]  reliable.”49 

Additional  causes  for  transfer  failure  were  postulated,  including  the  possibility  of  an 

ineffective  simulator  training  approach: 

The  reader  should  keep  in  mind  the  integral  relationship  of  training 
program  and  training  device.  Training  effectiveness  is  a  function  of  both 
of  these.  Even  though  significant  training  effectiveness  was  not 
demonstrated  under  the  conditions  of  the  current  study,  this  does  not  mean 
that  all  SAAC  training  will  be  noneffective.  It  only  means  that  the  right 
training  program  and  training  device  capability  combination  was  not 
demonstrated  in  this  study.50 

Instructors  “were  reluctant  to  use  SAAC  much  differently  than  they  might  employ  an 
aircraft,  that  is,  they  tended  to  ignore  many  of  the  unique  instructional  features  of  the 
SAAC.”51  For  example,  instructors  did  not  use  the  simulator  freeze  or  reset  capability  to 
identify  and  fix  errors  as  they  occurred;  instead,  errors  were  allowed  to  compound. 


1980  SAAC  TOT  Study 

Another  SAAC  TOT  study  was  conducted  during  an  F-4E  Fighter  Weapons 
Instructor  Course  (FWIC)  in  1980.  This  time  more  objective  measures  were  used,  and 
successful  transfer  was  observed  during  BFM  and  ACM  simulator  training.  “SAAC- 
trained  pilots  had  significantly  more  valid  missile  and  gun  shots.  They  also  achieved 
higher  exchange  ratios  and  achieved  a  higher  class  standing  in  the  course.”  ~  The  results 
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were  statistically  “significant,”  but  barely  so;  the  differences  in  performance  between  the 

ci 

simulator-trained  group  and  the  control  group  were  still  relatively  small. 

Unfortunately,  no  air-to-air  combat  simulator  TOT  studies  have  been  documented 
since  1980.  This  represents  a  break  of  almost  20  years,  with  no  TOT  studies  of  multi¬ 
ship  simulation. 


Summary  of  Simulator  Study  Findings 

Although  in  some  cases  the  results  were  only  slight,  overall  the  utility,  ISL,  and  TOT 
studies  provide  confidence  that  air-to-air  simulator  training  can  be  effective.  Review  of 
these  studies  uncovered  several  findings  that  may  be  applicable  to  modem  simulation 
training  such  as  DMT.  These  include  some  insight  into  the  benefits  and  limitations  of  air 
combat  simulation  training,  as  well  as  offering  techniques  to  improve  programs  to 
maximize  their  training  potential.  The  next  chapter  will  discuss  the  implications  of  these 
findings  for  improving  F-15C  air  combat  training  with  DMT. 
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Chapter  3 


Implications  for  DMT 

We’re  going  to  have  to  prove  to  them  that  it  works,  and  the  measure  of 
merit  will  be  when  those  F-15  crews  come  out  of  those  simulators  in  1999, 
sweating  and  with  smiles  on  their  faces.  That  will  happen  because  they 
will  have  gotten  high  quality  training  ...  they  simply  can’t  get  anywhere 
else. 

— General  Richard  E.  Hawley 

Training  effectiveness  studies  provide  some  insight  for  improving  air  combat 
simulator  training.  Findings  demonstrated  simulator  training  was  effective,  while 
highlighting  some  of  the  limitations  and  unique  benefits  of  such  training  and  offering 
some  techniques  to  improve  simulator  training  programs.  These  findings  provide  insight 
on  how  to  better  prepare  F-15C  pilots  for  air  combat  through  Distributed  Mission 
Training. 

Of  the  ten  simulator  studies  reviewed,  all  but  one  at  least  confirmed  air  combat 
simulation  training  is  effective,  contributing  to  pilot  learning  and  improving  performance. 
The  single  study  not  showing  positive  learning  was  at  worst  neutral,  so  it  does  not 
necessarily  belie  the  value  of  simulator  training — it  simply  did  not  find  a  measurable 
difference  in  pilot  flight  performance  due  to  simulator  training.  The  MUFTIRAD  ISF 
study  and  most  of  the  TOT  studies  demonstrated  only  small  positive  effects.  This  may 
highlight  limitations  of  the  metrics  used,  the  training  devices,  the  training  programs,  or  all 
of  the  above.  The  question  of  training  device  could  have  been  resolved  if  training 
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accomplished  in  flight  was  shown  to  improve  performance  while  simulator  training  did 
not,  but  no  such  comparisons  were  made. 

None  of  the  results  compared  simulator  training  to  in-flight  training.  Although  the 
MACAIR  utility  study  used  pilot  surveys  to  identify  tasks  perceived  as  better  trained  in 
the  simulator,  there  were  no  measured  performance  differences  between  simulator  and  in¬ 
flight  training.  The  Navy  TOT  study  was  the  only  study  even  suggesting  such  a 
comparison,  but  it  was  unable  to  actually  measure  a  demonstrable  difference  in  flight 
performance. 

There  were,  none-the-less,  many  lessons  learned  that  can  be  applied  to  DMT 
programs.  Some  of  the  results  presented  were  nothing  more  than  conjecture,  but  should 
at  least  be  considered  for  DMT  program  development.  Unfortunately,  most  of  the  studies 
were  not  successive  or  collective,  so  none  of  the  results  led  to  deeper  investigation. 
Although  findings  do  not  provide  startlingly  new  evidence  for  simulator  training 
application,  they  do  highlight  several  concepts  that  should  be  applied  in  DMT  programs 
to  maximize  training  effectiveness. 

Unique  Benefits  of  Simulation 

The  MACAIR  utility  study  surveys  revealed  several  important  advantages  of 
simulation  training  over  in-flight  training.  These  included  instruction  options  unique  to 
simulator  training,  the  ability  to  repeatedly  practice  desired  training  scenarios,  and  the 
opportunity  to  train  in  dangerous  and  complex  mission  scenarios  with  real-time  kill 
removal. 
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Instruction  Options 

DMT  offers  several  instruction  options  not  possible  in  flight  training.  These  options 
can  easily  be  ignored  as  pilots  default  to  their  usual  flight  instructional  techniques.  The 
1977  SAAC  TOT  study  found  instructors  “were  reluctant  to  use  [simulators]  much 
differently  than  they  might  employ  an  aircraft,  that  is,  they  tended  to  ignore  many  of  the 
unique  instructional  features  of  the  [simulator].”1 

During  flight  training,  resetting  an  engagement  expends  valuable  time  and  fuel,  and 
instructors  must  wait  until  after  the  debrief  to  provide  detailed  feedback.  In  addition, 
pilots  must  wait  for  another  sortie  to  fix  their  errors.  Distributed  Mission  Training,  on  the 
other  hand,  will  allow  instructors  to  “coach”  players  during  air  combat  scenarios — an 
instructor  can  observe  performance,  freeze  the  mission,  provide  guidance,  then  either 
resume  or  reset  the  scenario.  This  will  also  allow  other  “extrinsic  feedback,”  such  as 
“cueing”  (preparing  a  trainee  to  look  for  rare  events  or  signals  typically  ignored  just 
before  they  are  presented),  and  “prompting”  (presenting  the  correct  response  immediately 
after  a  stimulus  or  event  occurs).'  If  desired,  freeze,  reset,  and  replay  features  are  also 
selectable  from  the  simulator  cockpits.  This  unique  DMT  capability  should  not  be 
overlooked. 

DMT  scenarios  can  also  be  digitally  recorded  and  saved  for  later  playback  in 
simulator  cockpits.  F-15C  Flying  Training  Unit  (FTU)  instructors  at  Tyndall  AFB  are 
using  this  feature  with  their  simulators  to  introduce  student  pilots  to  various  sight  pictures 
and  situations,  like  a  360-degree  Omni-Max  theater.4  Operational  units  should  also 
consider  using  this  capability  to  instruct  more  advanced  tactical  situations  in  DMT.  This 
could  be  used,  for  example,  to  help  pilots  distinguish  between  various  AIM-9M  tones 
while  tracking  a  target  with  flares,  or  teach  pilots  to  visually  recognize  whether  a  missile 
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in  flight  has  failed  or  is  tracking  a  target  (i.e.  with  appropriate  lead  during  intercept). 
After  all,  “a  picture  is  worth  a  thousand  words  (flights)....  Visual  pictures  provide  the 
best  learning  tool  for  comprehension,  retention  and  reinforcement.”5 

Finally,  DMT  will  provide  extremely  detailed  debriefing  capabilities  not  available  in 
flight  training.  Each  mission  will  be  digitally  recorded  and  can  be  replayed  on  large 
screens,  showing  a  “God’s  eye”  view  of  the  entire  mission,  similar  to  Air  Combat 
Maneuvering  Instrumentation  (ACMI).  Cockpit  views  will  also  be  available  and  can 
include  selectable  overlays  of  aircraft  radar,  HUD,  and  Tactical  Electronic  Warfare 
System  (TEWS)  displays.  These  are  but  a  few  of  many  options  available  to  accurately 
recreate  and  analyze  each  mission.6 

Practice 

Another  unique  training  benefit  of  simulation  is  the  opportunity  to  practice. 
Simulators  not  only  provide  pilots  with  additional  “flight  time,”  but  they  can  also  isolate 
specific  tasks  for  repeated  exposure,  allowing  pilots  to  practice  skills  in  need  of  focused 
training.  Practicing  achieves  “overlearning,”  which  improves  the  chances  of  retaining 

7 

task  skills  and  habits,  and  “makes  performance  of  the  task  more  resistant  to  stress.” 
Practice  and  overlearning  can  become  dangerous,  however,  if  they  develop  undesirable 
“side-effects”  or  habits  (negative  training).  After  all,  “practice  does  not  make  perfect, 

o 

practice  makes  permanent.” 

Distributed  Mission  Training  will,  for  the  first  time,  provide  operational  F-15C  units 
with  the  capability  to  isolate  and  repeatedly  practice  team  skills.  Before  DMT,  F-15C 
simulator  training  was  limited  to  single-ship  operations  that  could  cause  negative  training 
by  not  allowing  pilots  to  develop  team  habits  required  in  actual  flight.  Practicing 
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targeting  in  an  isolated  simulator,  for  example,  can  build  a  single-ship  mindset  and  a 
habit  of  “heads  down”  operation,  ignoring  important  communication  and  visual  lookout 
requirements  during  this  crucial  phase  of  an  intercept.9  With  DMT,  pilots  can  practice  as 
they  would  in  flight — as  a  team.  This  will  also  provide  flight  leads  with  more  exposure 
to  decision-making  opportunities. 

Complex  Mission  Scenarios 

Pilots  surveyed  during  the  MACAIR  utility  evaluations  considered  mission  scenarios 
to  be  the  “key  ingredient  to  the  training  value”  of  multiship  simulator  training.10 
Similarly,  DMT  scenarios  can  be  customized,  allowing  pilots  to  practice  a  wide  variety  of 
missions.  Real-world  airborne  and  surface-to-air  threats  will  be  selectable  to  provide 
realistic  dissimilar  air  combat  training  (DACT)  not  readily  available  in  flight  training.11 

Rigorous  DMT  scenarios  can  prepare  pilots  for  complex  training  exercises  such  as 
Red  Flag  or  Cope  Thunder.  This  was  demonstrated  in  a  distributed  mission  simulation 
study  involving  A- 10  aircraft  conducting  close  air  support  (CAS)  missions.  Practicing  a 
Red  Flag  scenario  in  distributed  simulators  “did  well  to  prepare  young  wingmen  for  their 
first  real  Red  Flag,”  dealing  with  the  complicated  communications  and  timing  for 
multiple  aircraft  strikes.  ~  Even  more  importantly,  DMT  complex  training  should  prepare 
pilots  for  actual  combat. 

DMT  offers  another  unique  feature  of  combat  not  possible  in  flight  training:  real¬ 
time  kill  removal.  The  MACAIR  utility  evaluations  identified  this  as  “an  especially 
important  advantage  of  simulator-based  training.”  This  capability  is  particularly  helpful 
against  multiple  threats,  allowing  untargeted  threats  to  be  sorted  after  killed  threats 
disappear.  In  addition,  “killed”  F-15C  pilots  will  receive  immediate  performance 
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feedback  from  poorly  executed  tactics  when  their  simulator  ceases  to  function,  indicating 


their  death. 

High  risk  tasks  can  also  be  practiced  in  the  DMT  environment  without  jeopardizing 
pilot  safety.  A  typical  example  is  practicing  emergency  procedures;  but  with  DMT, 
realistic  visual  displays  can  include  dangerous  tactics  such  as  low-altitude  defensive 
maneuvers  against  SAMs  or  artillery.  However,  pilots  should  be  aware  that  lacking  “fear 
of  demise”  might  build  habits  of  performing  maneuvers  unsafe  to  accomplish  in  actual 
flight.14 


Training  Device  Limitations 

Sources  of  Negative  Training 

Because  simulators  cannot  duplicate  every  aspect  of  flight,  pilots  have  long  been 
concerned  with  negative  training.  Pilots  have  often  been  required  to  put  up  with  various 
unrealistic  aspects  of  simulation,  commonly  referred  to  as  “sim-isms.”  “Many  examples 
of  negative  training  have  been  demonstrated  in  flight  simulator  work....  It  is  very 
important  to  be  able  to  identify  elements  of  training  which  can  lead  to  negative 
transfer.”15  An  evaluation  of  negative  training  sources  identified  in  previous  studies  may 
highlight  negative  factors  possibly  affecting  DMT.  Once  potential  sources  of  negative 
training  are  identified,  the  challenge  remains  to  determine  how  to  minimize  their  effects. 

Surprisingly,  the  MACAIR  utility  study  was  the  only  one  to  address  potential 
sources  of  negative  training.  A  majority  of  the  problems  identified  were  concerned  with 
specific  hardware  and  software  limitations  of  the  MACAIR  facility.  Problems  included 
target  projection  colors  and  resolution,  cockpit  hardware  differing  from  F-15C  aircraft, 


30 


unrealistic  WD  information,  and  unrealistic  air  threats.  Fortunately,  F-15C  MTC 
simulators  are  expected  to  be  much  more  sophisticated  than  the  1990  MAC  AIR 
simulators — cockpit  hardware,  visual  fidelity,  WD  stations,  and  threats  should  be  much 
more  realistic. 

Pilots  in  the  MACAIR  utility  study  were  confused  by  target  aircraft  color  changes, 
and  complained  “that  aircraft  attitude,  aspect  angle,  and  tactical  range  could  not  be 
identified  within  distances  typical  in  the  air  ...  [interfering]  with  tactical  formation,  visual 
mutual  support,  BFM,  and  ACM/ACT.”16  The  F-15C  MTC  cockpit  high  resolution 
Target  Projection  System  (TPS)  will  project  a  single  “monochrome  green”  color  to 
represent  target  images.  It  will  “provide  sufficient  resolution  to  determine  target  aspect 
and  rate  changes  of  an  F-15  sized  aircraft  up  to  12,000  feet”  allowing  normal 
formation-keeping.  It  will  display  up  to  seven  targets  in  high  resolution  from  a  realistic 
tally  range  of  ten  nautical  miles  to  as  close  as  300  feet.  These  capabilities  should 
prevent  target  projection  problems  identified  in  the  MACAIR  study. 

Pilots  also  found  problems  adapting  to  a  modified  F-15E  cockpit.  F-15C  MTCs  will 
prevent  hardware  mismatches  by  ensuring  all  simulator  cockpits  are  fitted  with  current 
aircraft  hardware  and  software.  By  contract  “the  system  will  be  kept  current  with  the  F- 
15C  to  include  all  Operational  Flight  Programs  (OFP)  and  any  and  all  modifications 
scheduled  for  inclusion  into  the  aircraft.”19  Upgrades  will  be  incorporated  “within  the  60 
day  period  preceding  the  operational  use  of  modified  or  upgraded  home  station 
aircraft.”20 

Unrealistic  WD  information  was  previously  eliminated  by  MACAIR  after  their 
utility  evaluation  by  incorporating  a  more  realistic  WD  station.'  MTC  weapons  director 
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stations  should  also  be  acceptable,  designed  “to  allow  a  controller  to  provide  realistic 
radar  control  to  the  scenario.”  (Emphasis  added)22 

Also  mentioned  was  pilot  dissatisfaction  with  constructive  and  interactive  threat 
behavior.  Constructive  threats  seemed  to  operate  beyond  the  maneuvering  or  weapons 
capabilities  of  actual  threats,  with  superb  capability.  The  MTC  combat  environment,  will 
“accurately  model  physical,  electronic,  and  aerodynamic  capabilities  of  real  world  threats 
including  ...  flight  and  maneuvering  performance... sensors... on-board  weapons  ... 
evasive  maneuvers,”  and  other  threat  specific  capabilities.  Threats  will  also  “emulate  real 
world  tactics  and  doctrine  based  upon  ...  the  identified  experience  level  of  the  crew.”' 
This  will  allow  pilots  to  select  scenario  difficulty  by  choosing  the  performance  capability 
of  their  opponents. 

Manned  interactive  threats,  controlled  by  F-15  pilots  from  an  adversary  cockpit,  “did 
not  provide  a  challenging  or  credible  threat.”'  This  was  partially  due  to  F-15  pilots’  lack 
of  proficiency  using  adversary  cockpit  displays  and  controls.  MTC  threat  cockpits  will 
also  be  unique — controls  and  displays  will  not  resemble  those  in  the  F-15.  Having  F-15 
pilots  fly  as  Red  Air  in  these  cockpits  could  not  only  reduce  threat  credibility  for  training 
Blue  Air  pilots,  but  could  also  cause  negative  training  for  pilots  operating  the  unfamiliar 
devices.  As  an  alternative,  any  of  the  four  F-15  simulators  could  be  used  to  replicate 
threats. 

Fortunately,  F-15  pilots  are  not  expected  to  operate  MTC  interactive  threat  cockpits. 
The  MTC  concept  of  operations  calls  for  contractor  instructors  to  operate  threat  stations 
rather  than  other  F-15C  pilots.'  This  concept  is  encouraging,  since  it  should  also  reduce 
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requirements  for  F-15  pilots  to  fly  adversary  tactics  (which  could  arguably  be  another 
source  of  negative  training  for  F-15  pilots). 

While  MTC  design  specifications  seem  to  rule  out  identified  sources  of  negative 
training  in  past  simulator  facilities,  a  test  of  the  working  system  will  be  required  for 
confirmation.  When  the  first  F-15C  MTC  becomes  functional  this  spring,  a  simulator 
certification  (SIMCERT)  process  will  utilize  aircrew  expertise  to  ensure  its  simulators 
“are  physically  and  functionally  maintained  to  the  design  configuration  and  provide 
accurate  and  credible  aircrew  training....  Concurrently,  SIMCERT  [will  identify] 
potential  negative  training  due  to  inaccurate  simulation  of  aircraft  systems  or 
performance  characteristics.”'  Hopefully  MTCs  will  meet  or  exceed  all  design 
specifications.  (Other  planned  MTC  evaluations  are  discussed  in  appendix  C.) 

Physiological  Considerations 

Conspicuously  absent  from  identified  sources  of  negative  training  is  the  inability  to 
simulate  physiological  factors  present  in  flight."  Simulator  studies  investigating 
physiological  issues  do  exist,  but  none  were  found  addressing  implications  for  air  combat 
training.  Although  the  1977  SAAC  TOT  study  did  attempt  to  measure  affects  of 
simulator  motion,  the  results  were  inconclusive,  neither  supporting  nor  discouraging  use 
of  simulator  motion. 

Flying  exposes  aircrew  to  other  stresses  not  possible  to  simulate:  physiological  stress 
such  as  Gs,  altitude,  and  stress  from  the  realization  that  a  mistake  in  flight  can  cause  loss 
of  life.  Pilots  must  get  enough  in-flight  training  to  acclimatize  themselves  to  these 
stresses,  to  build  good  anti-G  straining  maneuver  (AGSM)  habits  and  techniques,  28  to 
learn  aircraft  handling  characteristics  and  “feel,”  and  to  make  tough  decisions  in  real-life 
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scenarios.  How  much  and  how  often  pilots  need  to  “shake  and  bake”  to  be  acclimatized 


•  •  29 

remains  uncertain.' 


Training  Programs 

Knowledge  of  the  intrinsic  benefits  and  limitations  of  DMT  can  serve  as  the  basis  for 
developing  an  effective  training  program.  The  most  obvious  example  is  the  need  to 
develop  programs  avoiding  or  reducing  the  effects  of  negative  training  when  they  are 
unavoidable. 

Reducing  the  Effects  of  Negative  Training 

Even  if  DMT  simulation  does  not  perfectly  reflect  reality,  this  should  not  prevent  a 
well-designed  training  program  from  being  effective.  Simulator  fidelity  is  often 
inappropriately  blamed  for  training  ineffectiveness — doing  so  “ignores  the  manner  in 
which  a  device  is  used  and  the  objectives  of  device  training.  These  two  considerations 

TO 

underlie  any  determination  of  simulator  training  effectiveness.” 

The  underlying  objective  of  each  task  or  training  event  needs  to  be  considered  for 
determining  the  appropriateness  of  accomplishing  that  training  in  either  flight  or  with 
DMT.  There  is  a  “need  for  a  proper  task  analysis  and  an  understanding  of  the  learning 
processes  associated  with  achieving  a  satisfactory  performance  of  the  task,  before 
determining  the  nature  of  the  training  equipment  to  be  employed.”  To  use  a 
controversial  example,  if  the  BFM  objective  is  to  learn  energy  management  by  honing 
pilot  sensitivity  to  aircraft  Gs  and  buffet  cues,  this  objective  must  be  accomplished  in 
flight.  On  the  other  hand,  if  BFM  objectives  are  to  recognize  threat  energy  state  based  on 
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visual  cues  such  as  closure  and  line-of-sight  rate,  DMT  can  be  very  useful — the  picture 
can  even  be  frozen  or  replayed  for  emphasis. 

Training  events  are  actually  more  complex  than  this  and  often  cannot  be  isolated  to  a 
single  objective  or  single  “cue.”  Still,  flight  leads  and  instructors  can  exploit  the 
capabilities  of  each  device  (simulator  or  aircraft)  by  emphasizing  cues  the  training  device 
is  most  capable  of  accentuating.  Combining  both  devices  will  provide  the  best 
opportunity  to  satisfy  most  training  objectives.  Using  another  BFM  example, 
“demonstrating  the  effects  of  increased  turn  rates  without  the  high-Gs  in  the  simulator 
may  enhance  the  pilot’s  conceptual  understanding  to  be  applied  to  the  mission  the  next 
day.”32 

For  those  aspects  of  flight  DMT  cannot  simulate,  the  potential  for  negative  training 

is  unavoidable.  However,  it  may  be  possible  to  reduce  the  risk  of  negative  training 

through  proper  instruction.  The  1975  Navy  TOT  study  mentioned  the  “powerful 

technique  of  ...  training  ‘around’  (or  in  spite  of)  limitations  that  are  inherent  in  the 

simulator.”  This  concept  is  also  described  in  an  AFHRL  guide  for  conducting  TOT 

studies:  “Studies  showed  that,  if  the  problem  is  made  quite  clear  to  the  student  prior  to 

and  during  simulator  work  and  prior  to  and  during  airborne  work,  such  training  for 

transfer  completely  offsets  the  potential  [of  negative  transfer],  the  student  having  little 

difficulty  in  either  the  simulator  or  the  aircraft.”  (Emphasis  in  original)34  In  other  words, 

simply  reminding  pilots  of  what  the  actual  environment  entails  can  reduce  the  effects  of 

negative  training.  If  the  potential  for  negative  transfer  is  high,  more  frequent  reminders 

may  be  required  to  ensure  pilots  maintain  a  realistic  mindset. 

This  procedure  need  not  be  paradoxical  because,  frequently,  different 
pilots  make  use  of  different  sets  of  cues  as  aids  during  performance  of  the 
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same  maneuver;  these  perhaps  depending  on  their  individual  preferences. 

Even  the  same  pilot  may  use  different  sets  of  cues  at  different  times.... 

-5  C 

The  pilot  makes  do  with  alternatives  that  serve  the  same  purpose. 

In  addition  to  identifying  sources  of  negative  training  and  reducing  their  effects,  air 
combat  simulation  studies  provide  additional  insight  for  improving  simulator  training 
programs.  Findings  suggest  how  to  exploit  DMT’s  unique  training  capabilities  by 
identifying  what  tasks  should  be  emphasized  in  DMT,  who  will  gain  the  most  training 
benefit  from  DMT,  and  when  to  schedule  DMT  events  to  optimize  training. 

Tasks  to  Emphasize  in  DMT 

The  most  obvious  tasks  to  emphasize  in  DMT  are  those  tasks  or  missions  incapable 
of  being  trained  in  flight,  such  as  the  dangerous  tasks  or  complex  missions — DMT 
provides  a  unique  benefit  for  this  training.  More  generally,  DMT  should  focus  on 
continuation  training  (CT)  shortfalls — tasks  needing  more  attention  than  is  provided  in 
flight  training. 

MACAIR  utility  evaluations  provide  a  list  of  combat  tasks  considered  best  trained  in 
the  simulator,  correlating  with  tasks  in  need  of  additional  training  (table  1).  Although 
this  specific  list  is  likely  outdated,  it  demonstrates  the  concept  of  focusing  DMT  on  CT 
shortfalls.  These  shortfalls  may  vary  each  year,  and  may  even  vary  by  unit,  since  training 
opportunities  and  restrictions  differ  for  each  wing.  The  ACC  commander  also  envisioned 
DMT  as  a  tool  to  compensate  for  CT  shortfalls.  General  Hawley  recognizes  air  combat 
training  becoming  increasingly  constrained  due  to  shrinking  budgets,  airspace  limitations, 
concern  for  safety,  and  operational  tasks  limiting  training  opportunities.36 

Even  if  some  tasks  or  mission  segments  are  not  initially  viewed  as  shortfalls  because 
they  are  included  in  CT,  they  may  still  require  extra  attention  simply  because  in-flight 
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training  does  not  provide  sufficient  time  to  reach  a  desired  level  of  proficiency.  For 
example,  only  seconds  of  training  are  provided  during  an  entire  flight  for  conducting 
threat  identification  in  accordance  with  established  ROE  and  employing  weapons.  DMT 
will  allow  teams  to  quickly  reset  and  repeat  any  mission,  mission  segment,  or  task  to  gain 
“psychomotor”  time  exercising  desired  skills  to  become  proficient  as  a  team. 

Difficult  tasks  may  be  identified  as  CT  shortfalls  because  they  require  more  training 
to  master.  Simulation  allows  focused  exposure  to  difficult  tasks,  so  it  can  serve  as  an 
outstanding  tool  to  improve  performance.  The  1975  Navy  TOT  study  even  suggested 
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training  transfer  is  enhanced  when  a  training  task  is  more  difficult  than  the  actual  task. 
For  DMT  this  simply  means  training  should  be  as  challenging,  if  not  more  challenging, 
than  actual  flight  (or  actual  combat).  This  is  supported  by  pilots  surveyed  in  the 
MACAIR  utility  study,  who  thought  the  “challenging,  demanding,  [and]  realistic” 

on 

scenarios  were  “the  key  ingredient  to  the  training  value”  of  multiship  simulation. 
“Perishable  skills”  will  also  seem  difficult  if  proficiency  is  allowed  to  atrophy.  DMT  can 
provide  pilots  with  multiple  exposure  to  skills  to  regain  proficiency  or  currency  in  a 
“safe”  environment. 

Improving  Training  with  Metrics 

Units  can  identify  major  CT  shortfalls  intuitively  or  by  survey.  The  MACAIR  utility 
evaluations  found  remarkable  consistency  in  tasks  needing  additional  training.  However, 
using  performance  measures  to  identify  shortfalls  may  prove  more  useful  in  highlighting 
less  obvious  tasks  needing  focused  training.  The  same  metrics  can  provide  definitive 
data  for  identifying  trends,  and  provide  feedback  to  ensure  both  in-flight  training  and 
DMT  programs  improve  performance. 
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The  ISL  and  TOT  studies  offer  several  measurement  methods.  Many  of  the  studies 
used  “outcome”  measures,  identifying  only  the  final  result  of  an  engagement.  Outcome 
measures  may  be  useful  for  evaluations  or  competitions,  but  are  not  useful  for 
determining  what  caused  that  outcome.  “Process”  measures,  on  the  other  hand,  can 
identify  causal  effects.  This  can  be  extremely  useful  for  determining  what  training 
aspects  need  emphasis.40  The  1975  Navy  TOT  study  attempted  to  measure  how  mission 
outcomes  were  achieved  using  performance  variables  (PV).  Their  analysis  identified  two 
PVs  most  responsible  for  improving  rolling  scissors  maneuvers:  “Use/Control  of  Energy, 
and  Nose  to  Tail  Separation.”41 

The  1977  SAAC  TOT  study  “skills”  and  the  1993  MULTIRAD  study  SA 
“behavioral  indicators”  (table  2)  are  also  examples  of  process  measures.  They  look 
remarkably  similar  to  common  grade  sheets — breaking  down  missions  or  engagements 
into  discrete  tasks  leading  to  success.  Unfortunately,  neither  of  these  studies  analyzed 
their  data  to  determine  how  the  processes  contributed  to  each  mission  as  the  Navy  TOT 
study  did;  process  measures  were  used  only  to  derive  an  overall  performance  score  for 
each  engagement. 

For  both  outcome  and  process  measures  of  performance,  subjective  ratings  often  did 
not  produce  significant  variance  to  measure  trends,  such  as  the  instructor  ratings  used  in 
the  1977  SAAC  TOT  study.  ~  Instructors  tend  to  ignore  extreme  grades  on  scaled 
measurements,  “because  of  operational  practice  calling  for  a  majority  of  grades  to  cluster 
in  the  region  of  ‘average.’”43  The  1975  Navy  TOT  study  attempted  to  compensate  for 
this  practice  by  expanding  their  grading  to  a  12-point  scale,  but  this  still  largely  resulted 
in  clustered  grades — the  rolling  scissors  being  the  only  maneuver  demonstrating  a 
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significant  variance  (figure  5).  Objective  measures  used  in  the  first  three  ISL  studies  and 
the  1980  SAAC  TOT  study  were  more  successful  in  obtaining  significant  results,  using 
AAMI  or  weapons  employment  data. 

Future  efforts  to  measure  and  compare  both  simulator  and  flight  task  performance 
could  be  useful  for  several  purposes.  First,  it  could  identify  tasks  receiving  a  large 
training  benefit  from  simulator  training — DMT  could  then  be  used  as  a  primary  training 
tool  for  those  tasks.  Second,  tasks  receiving  little  training  benefit  could  be  directed  to 
other  training  means  (i.e.  swap  emphasis  from  DMT  to  flight  or  vice  versa) — this  would 
ultimately  determine  the  proper  mix  of  in-flight  and  DMT  sorties  needed  to  develop  and 
maintain  task  proficiency  at  desired  performance  criterion.  Third,  noting  changes  in  task 
performance  by  varying  setups,  scenarios,  and  instructor  techniques  can  help  identify 
methods  for  optimizing  these  variables  to  match  desired  objectives.  Finally,  the  same 
task  measures  can  be  used  to  compare  and  track  individual  pilot  abilities,  identifying  their 
specific  training  needs. 

Performance  measures — like  statistics  for  baseball  players — provide  a  record  of 
players’  capabilities,  provide  trend  information  to  monitor  progress  or  regression,  provide 
feedback  to  the  players,  stimulate  competition,  and  identify  individuals  for  awards.44 
Many  F-15C  squadrons  already  use  some  performance  measures,  such  as  weapons 
employment  data,  to  determine  monthly  “Top  Gun”  award-winners.  This  data  could  be 
used  to  identify  weapons  employment  skills  needing  extra  training — training  that  can  be 
emphasized  in  DMT.  Top  Gun  competition  could  easily  be  expanded  to  include  more 
than  just  weapons  data.  A  simple  example  is  tracking  the  percentage  of  times  a  pilot 
“targets”  the  correct  group.  Top  Gun  could  also  include  DMT  events  to  determine 
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simulator  training  performance.  This  would  provide  a  much  larger  representation  of 
training  engagements  accomplished,  and  motivate  pilots  in  the  simulator. 

Using  objective  performance  measures  to  customize  training  can  also  reduce  debates 
about  the  desired  complexity  of  training  profiles  for  pilots  of  varying  flight  qualifications, 
as  was  seen  in  the  MAC  AIR  utility  study.  Measuring  performance  would  determine 
whether  or  not  a  profile  is  in  fact  too  complex  or  “too  demanding  for  a  new  wingman.”45 

Customize  Training  Profiles  by  Flight  Qualification 

Personalizing  training  for  each  individual  pilot  may  be  difficult,  but  possible  using 
DMT  with  its  flexible  scenarios  and  setups.  However,  for  planning  purposes,  operational 
units  need  to  design  training  profiles  in  advance,  from  which  modifications  can  be  made 
for  individuals.46  Finding  logical  “audience”  groups  would  be  extremely  useful  to 
determine  how  to  vary  tasks  and  missions  in  “customized”  profiles. 

The  1993  MULTIRAD  ISL  study  demonstrated  air  combat  simulation  was  most 
beneficial  for  (or  most  appreciated  by)  upgrading  pilots  (figure  4).  Also,  new  wingmen 
in  the  MACAIR  utility  study  (who  can  be  considered  “upgrading”  pilots)  found  training 
more  valuable  than  did  experienced  pilots  in  several  tasks.47  This  concept  is  not 
inconsistent  with  difficult  or  perishable  skills  requiring  extra  training,  since  upgrading 
pilots  are  likely  to  find  new  flight  duties  or  tasks  difficult  due  to  their  inexperience. 
DMT  now  offers  the  capability  to  expose  upgrading  or  inexperienced  flight  leads  to 
multiple  “difficult”  situations,  allowing  them  to  develop  and  exercise  their  new  decision¬ 
making  responsibilities. 

Currently  CT  requirements  vary  according  to  pilot  experience  (total  flight  hours),  as 
reflected  in  the  Ready  Aircrew  Program  (RAP).  In  determining  how  to  vary  MTC 
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training  profiles,  it  may  be  useful  to  further  divide  training  plans  by  flight  qualification, 
since  required  tasks  change  dramatically  for  each  level  (wingman,  two-ship  lead,  four- 
ship  lead,  and  IP).  While  refining  the  SA  rating  scales  (SARS),  an  Armstrong  Laboratory 
study  discovered  average  pilot  SA  scores  increased  significantly  with  each  increase  in 
flight  qualification,  accounting  for  68%  of  the  variance  (figure  7).  This  is  profound, 
considering  F-15  hours  only  accounted  for  35%  of  the  variance,  and  total  flight  hours 
only  15%.48 


Flight  Qualification 


Figure  7.  Composite  SA  Score  as  a  Function  of  Flight  Qualification49 

After  all,  “pilots  do  not  get  the  amount  of  practice  at  each  skill  level  that  would  be 
required  for  them  to  achieve  their  full  potential.”  (Emphasis  in  original)50  While 
developing  training  plans  for  DMT  (or  even  new  RAP  requirements),  it  may  prove  useful 
to  search  for  training  tasks  needing  emphasis  at  each  flight  qualification  level. 


Scheduling  DMT  Events 

The  1975  Navy  TOT  study  postulated  two  “diluting  effects”  reduced  the 
effectiveness  of  simulator  training.  One  of  the  diluting  effects  was  caused  by  mass 
training  all  simulator  events  before  in-flight  training  began.  Researchers  believed 
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sequencing  in-flight  events  to  immediately  follow  corresponding  simulator  events  would 
improve  training  transfer.  The  second  diluting  effect  was  an  excessive  time  delay 
between  simulator  training  and  in-flight  training,  essentially  allowing  pilots  to  “forget” 
what  they  learned,  or  lose  proficiency.  To  avoid  these  diluting  effects,  DMT  and  flight 
missions  with  matching  objectives  should  be  paired  and  scheduled  in  sequence.  This  way 
DMT  can  prepare  pilots  for  the  subsequent  flight  by  exercising  skills  or  tasks  needing 
practice. 

The  concept  of  pairing  complimentary  DMT  and  flight  missions  resembles  formal 
training  syllabuses,  but  seems  much  less  typical  of  CT,  which  enjoys  scheduling 
flexibility.  None-the-less,  if  it  is  possible  to  do  the  same  in  CT,  pairing  the  two  training 
methods  will  theoretically  enhance  the  effectiveness  training.  Doing  so  may  also  allow 
pilots  to  reduce  briefing  and  debriefing  times,  saving  valuable  pilot  duty  hours  and 
increasing  time  efficiency. 

Once  the  AEF  cycles  begin,  units  returning  from  deployments  may  have  to  formalize 
CT  to  regain  combat  readiness.  Training  is  minimal  on  deployments,  so  when  units 
return  much  time  is  dedicated  to  regaining  currencies  and  upgrade  training,  leaving  less 
time  to  hone  complex  combat  skills.51  DMT  has  the  potential  to  add  efficiency  to 
syllabuses  by  providing  focused  training  and  rapid,  multiple  exposure  to  skills  needing 
emphasis.  This  would  allow  pilots  to  progress  more  quickly  from  fundamentals  back  to 
complex  scenarios,  both  in  the  air  and  in  DMT.  Once  multiple  aircraft  and  weapons 
systems  incorporate  DMT,  the  final  stage  of  training  can  concentrate  on  “spin-ups,” 
combining  AEF  teams  via  their  MTC  networks  to  train  together  in  complex  scenarios, 

CT 

and  conduct  mission  rehearsals  in  contingency  location  environments. 
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Chapter  4 


Improving  Future  Training 


If  DMT  and  its  inherent  value  is  appreciated  quickly,  then  today’s 
( current )  and  tomorrow’s  (future)  warriors  will  step  into  potenticd  battles 
with  the  great  advantage  of  improved  flying  ability,  reflexes  and  “seeing” 
the  big  picture  while  making  their  own  luck  along  the  way.  DMT  has  a 
future  and  is  destined  to  be  the  cornerstone  of  cur  and  space  dominance  in 
the  future. 


— Michael  R.  Oakes 


Summary 

Distributed  Mission  Training  can  improve  air  combat  training  for  F-15C  pilots  if 
smartly  integrated  into  continuation  training.  A  proper  balance  of  simulation  and  flight 
training  should  exploit  the  unique  benefits  of  each  training  method  while  avoiding  or 
reducing  their  limitations.  Previous  simulator  studies  provided  some  insight  into  the 
benefits  and  limitations  of  air  combat  simulation  training,  as  well  as  offering  techniques 
to  improve  programs  to  maximize  their  training  potential.  Most  of  these  findings  are 
applicable  to  DMT,  offering  combat  units  a  basis  for  developing  their  own  training 
programs  as  they  enter  the  DMT  network. 

Unique  Benefits  of  DMT 

Simulator  training  offers  several  benefits  not  available  in  flight  training,  including 
unique  instruction  options,  the  capability  to  repeatedly  practice  desired  tasks  or  missions, 
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and  the  opportunity  to  train  in  complex  scenarios  not  readily  available  in  flight.  DMT 
will  allow  instructors  to  “coach”  and  provide  extrinsic  feedback  such  as  cueing  and 
prompting.  Debriefing  can  be  conducted  “on  the  fly”  using  freeze,  replay,  and  reset 
features,  so  pilots  can  immediately  re-attempt  mission  challenges  and  correct  their  errors. 
DMT  will  also  offer  extremely  accurate  reconstruction  of  missions  for  debriefing 
purposes  in  a  wide  variety  of  formats  rarely  (if  ever)  available  in  flight  training. 
Instructors  can  even  introduce  pilots  to  new  concepts  or  demonstrate  complex  tactics 
using  MTC  simulators  like  an  “Omni-Max”  theater. 

Simulation  also  offers  the  capability  to  practice — not  only  through  providing 
additional  “flight  time”  for  pilots,  but  also  providing  focused  training  for  practicing 
specific  tasks  in  need  of  additional  training.  Repeatedly  practicing  tasks  achieves 
overlearning  for  tasks  that  need  to  become  second  nature. 

Complex  scenarios  not  easily  orchestrated  for  flight  training  are  very  easy  to  design 
and  conduct  through  simulation.  DMT  can  create  any  environment  with  a  multitude  of 
realistic  ground  and  air  threats  to  better  prepare  pilots  for  air  combat.  Real-time  kill 
removal  will  provide  pilots  with  immediate  and  realistic  feedback  during  each  mission. 
Training  for  high  risk  missions  or  dangerous  tasks  can  also  be  accomplished  without 
jeopardizing  pilot  safety. 

DMT  Limitations 

The  privilege  of  simulation  does  not  come  without  some  drawbacks,  however. 
Simulator  studies  identified  several  limitations  for  air  combat  training  such  as  sources  of 
negative  training,  and  lack  of  physiological  stresses  or  inputs  that  may  be  desired  in 
training.  Fortunately,  none  of  the  sources  of  negative  training  identified  in  the  studies 
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will  be  a  factor  for  F-15C  MTCs,  which  are  expected  to  have  superb  visual  projectors, 
threat  behavior  algorithms,  cockpit  hardware,  and  realistic  weapons  director  radar 
screens.  On  the  other  hand,  MTC  simulators  will  not  provide  motion  or  other 
physiological  stimulus  and  stresses  that  may  be  required  to  achieve  desired  training 
objectives.  Pilots  must  leam  to  cope  with  the  harsh  environment  of  flight  in  fighter 
aircraft  and  acclimatize  themselves  to  stresses  such  as  Gs,  vibrations,  heat,  exhaustion, 
altitude,  and  facing  deathly  consequences. 

Ideally  all  sources  of  negative  training  would  be  avoided  by  matching  training 
objectives  with  the  training  tool  that  provides  all  required  stimulus  and  interactions.  This 
does  not  imply  that  all  training  should  be  conducted  in  flight,  since  flight  training  carries 
its  own  restrictions.  Whether  the  restrictions  are  due  to  safety,  budget,  airspace,  or 
adversary  constraints,  these  limitations  should  be  viewed  with  the  same  concern  as  “sim- 
isms”  because  they  may  not  expose  pilots  to  needed  training,  or  they  may  force  pilots  to 
train  other  than  they  would  actually  fight  in  combat.1  When  sources  of  negative  training 
are  unavoidable,  the  effects  of  negative  training  can  be  reduced  through  good  instruction, 
by  reminding  pilots  how  their  training  environment  differs  from  the  one  they  will  face  in 
actual  combat. 

Developing  Training  Programs 

With  an  understanding  of  DMT  benefits  and  limitations,  F-15C  units  can  begin  to 
develop  effective  training  programs.  Simulator  studies  present  some  guidance  toward 
this  effort,  offering  several  training  program  considerations  to  achieve  the  most  effective 
results  from  simulator  training.  Studies  suggested  which  tasks  should  be  emphasized  in 
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DMT,  how  metrics  can  identify  training  needs,  who  is  expected  to  gain  the  most  benefit 
from  DMT,  and  when  to  schedule  DMT  events  to  optimize  training. 

Tasks  to  emphasize  in  DMT  are,  quite  simply,  those  tasks  not  receiving  sufficient 
training  from  flight  sorties.  Tasks  expected  to  be  training  deficient  are  those  not 
accomplished  in  flight  training  due  to  their  danger  or  complexity,  while  other  tasks  may 
be  insufficiently  trained  in  flight  due  to  flight  restrictions  from  budget  constraints,  limited 
airspace,  or  operational  tasks.  Less  obvious  training  shortfalls  may  be  due  to  only 
fleeting  opportunities  during  a  sortie  to  practice  a  necessary  skill,  therefore  not  providing 
pilots  with  enough  psychomotor  time  to  fully  develop  proficiency.  Difficult  tasks  or 
perishable  skills  may  also  require  additional  training  time  to  master  or  more  frequent 
exposure  to  maintain  proficiency.  DMT  provides  a  unique  capability  to  focus  on  such 
skills  and  practice  them  as  required. 

Metrics  can  also  help  identify  tasks  in  need  of  additional  training.  Objectively 
measuring  pilot  or  team  performance  variables  (process  measurements)  can  identify 
which  variables  are  most  often  responsible  for  mission  failure,  highlighting  their  need  for 
additional  training.  Using  metrics  to  measure  both  DMT  and  flight  training  performance 
can  determine  which  training  device  is  best  suited  for  a  specific  training  objective.  The 
same  measures  can  be  used  to  improve  training  scenarios  and  programs.  Finally,  metrics 
can  help  identify  individual  pilot  or  team  training  needs.  (There  is  still  much  work  to  be 
done  in  developing  useful  metrics  for  these  purposes.  Appendix  C  discusses  this  topic  in 
greater  detail.) 

Upgrading  pilots  or  those  pilots  recently  upgraded  to  a  new  flight  qualification  were 
identified  in  simulator  studies  as  those  expected  to  gain  the  most  benefit  from  DMT.  The 
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ability  to  isolate  new  tasks  for  practice  and  to  provide  multiple  exposures  to  new 
decision-making  challenges  makes  DMT  a  logical  choice  for  maturing  new  flight  leads  or 
instructors.  Training  profiles  for  DMT  should  also  vary  for  each  flight  qualification  to 
focus  training  on  unique  responsibilities  at  each  level. 

Finally,  melding  the  two  training  methods  (DMT  and  aircraft)  into  a  cohesive 
training  program  requires  proper  scheduling  to  avoid  diluting  effects  in  learning. 
Diluting  effects  include  improper  sequencing  and  lengthy  delays  between  related  DMT 
and  flight  events.  To  gain  the  most  benefit  from  DMT,  tasks  and  missions  should  build 
in  a  logical  progression,  sequenced  with  similar  flights,  creating  a  synergy  to  increase  the 
effectiveness  of  both  DMT  and  flight  training  while  reducing  the  negative  effects  of  both. 

Final  Comments 

This  paper  focused  on  the  development  of  DMT  for  F-15C  pilots  at  the  operational 
unit  level.  Simulator  study  findings,  along  with  other  literature  concerning  simulator 
training,  were  used  to  collect  important  considerations  for  the  F-15C  community  to  be 
aware  of  as  they  incorporate  the  new  DMT  tool  as  part  of  their  air  combat  training 
program.  With  these  considerations  in  mind,  units  should  be  able  to  develop  programs 
that  exploit  the  potential  benefits  of  DMT,  while  avoiding  simulation  limitations 
discovered  in  the  past. 

DMT  simulation  presents  new  opportunities  for  air  combat  training.  MTC 
simulators  should  be  treated  neither  as  aircraft,  nor  as  typical  simulators — DMT  is 
unique,  having  capabilities  that  can  be  exploited  to  create  complete,  effective  training 
programs,  fostering  combat-ready  pilots. 
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Notes 


1  Dr.  Herbert  H.  Bell,  telephone  conversation  with  author,  19  November  1998. 
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Appendix  A 


F-15C  Mission  Training  Centers  (MTC) 

Advanced  Simulators 

F-15C  operational  units  will  be  first  in  the  CAF  to  incorporate  Distributed  Mission 
Training  (DMT)  advanced  simulators.  These  simulators  will  have  authentic  cockpit 
controls,  displays,  avionics,  weapons  systems,  ground  handling,  and  flight  modeling. 
Engine  type  will  also  be  selectable  to  match  variations  in  aircraft  configuration. 
Simulator  cockpits  will  have  Full  Field  of  View  (FFOV)  visual  displays,  providing  an 
“out-the-window”  cockpit  view  covering  360  degrees  laterally  and  135  degrees 
vertically.  The  display  will  even  project  an  image  of  the  F-15’s  wings  and  vertical  tails. 
Aircraft  details  will  provide  enough  aspect  and  distance  cues  to  fly  formation  with  a 
wingman  out  to  two  nautical  miles,  and  in  the  future  will  include  a  night  mode  for  Night 
Vision  Goggle  (NVG)  training  capability.  The  terrain  features  will  replicate  the  local 
flying  area  or  other  selected  real  world  regions,  derived  from  standard  Defense  Mapping 
Agency  products  and  services.  This  synthetic  environment  will  provide  not  only  visual 
inputs  to  the  pilot,  but  also  communication,  navigation,  and  electronic  inputs  to  operate 
all  F-15C  avionics  and  weapons  systems.1 
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MTC  Capabilities 


Advanced  simulators  will  be  housed  in  a  Mission  Training  Center.  The  F-15C 
MTCs  at  Eglin  AFB  and  Langley  AFB  will  each  have  4  simulators  that  can  be  run 
individually,  or  linked  for  two-,  three-,  or  four-ship  operation.-  An  Instructor/Operator 
System  (IOS)  will  control  the  links  and  mission  scenarios,  allowing  instructors  at  the 
console  to  insert  threats,  emergency  situations,  weapons  hit/miss  data,  atmospheric 
conditions,  and  more.  Selectable  private  communication  will  allow  instructors  to 
comment  to  all  or  selected  individual  pilots  in  the  simulators. 

The  MTC  can  generate  both  constructive  and  interactive  threats  for  its  training 
missions.  Constructive  threats  include  computer-controlled  aircraft,  air-to-air  missiles, 
surface-to-air  missiles,  and  electronic  systems.  These  threats  will  be  programmed  with 
real-world  capabilities  reported  by  the  Defense  Intelligence  Agency  (DIA)  to  provide 
realistic  radar  detection,  weapons  capabilities,  and  tactics.4  Several  interactive  threat 
simulator  stations  in  the  MTC,  which  have  very  simple  and  generic  cockpits,  can  be 
“flown”  by  additional  pilots  or  instructors  to  provide  human  input  to  threat  reactions  and 
decisions  for  more  realistic  training.  They  too  can  be  operated  individually  or  linked  into 
a  scenario  to  generate  multiple  interactive  threats.5  The  threats  they  represent  are 
selectable  from  the  IOS,  not  only  giving  opposing  pilots  the  opportunity  to  visually 
identify  the  threat  aircraft  in  close  combat,  but  also  appropriately  restricting  the  weapons 
and  radar  capabilities  of  the  interactive  threat  simulator  to  the  known  capabilities  of  the 
real-world  threat.6 

A  radar  intercept  control  station  will  also  be  installed  in  each  F-15C  MTC,  which  can 
be  reconfigured  to  represent  either  a  ground-based  or  AW  ACS  director  station.  A 
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trained  weapons  director  (WD)  or  a  stand-in  pilot  can  operate  the  station  to  provide 

o 

control  for  F-15C  pilots  during  each  mission. 

MTCs  will  also  include  comprehensive  briefing  and  debriefing  facilities.  In  fact,  the 
entire  mission  will  be  digitally  recorded  and  can  be  replayed  on  large  screens,  showing  a 
“God’s  eye”  view  of  the  entire  mission.  The  system  will  have  selectable  scale  and 
viewing  perspectives  to  show  terrain  features,  pilot  views  from  various  angles,  or  other 
cockpit  instrument  displays  for  any  selected  pilot.9  Aircraft  data  can  also  be  paired  to 
quickly  show  their  relative  altitudes,  airspeed  closure,  and  range  from  each  other — 
similar  to  Air  Combat  Maneuvering  Instrumentation  (ACMI).  In  addition,  each  pilot’s 
radar  and  heads-up-display  (HUD)  can  be  recorded  on  videotape  for  removal  from  the 
facility.10 


Notes 

1  Contract,  F33657-97-D-0025  Exhibit  1,  “System  Specifications  for  F-15C  MSIP 
Simulation  Service,”  14  July  1997,  revised  5  September  1997,  4-17. 

2  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  F-15C  Mission  Training  Centers,”  Working  Draft,  29 
September  1998,  29. 

3  Contract,  18-22. 

4  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  Distributed  Mission  Training,”  13  October  1998,  15. 

5  HQ  ACC/DOTO,  “Concept  of  Operations  for  F-15C  Mission  Training  Centers,” 

11. 

6  Contract,  26-30. 

7  Ibid.,  30-31. 

8  HQ  ACC/DOTO,  “Concept  of  Operations  for  F-15C  Mission  Training  Centers,” 

11. 

9  Air  Combat  Command,  “Systems  Requirement  Document  (SRD)  for  the  CAF  F- 
15A/C  MSIP  Four-ship  Simulator  System  and  Aerial  Combat  Enhanced  Simulation 
(ACES)  Center,”  28  March  1997,  8. 

10  Contract,  23-25. 
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Appendix  B 


Expansion  of  DMT 

ACC  intends  to  incrementally  build  upon  the  DMT  base,  linking  the  first  two  F-15C 
MTCs  at  Eglin  AFB  and  Langley  AFB  with  an  AW  ACS  MTC  at  Tinker  AFB.  The  three 
MTCs  will  be  networked  through  a  Distributed  Warfighting  Center  (DWC)  “hub.”  The 
DWC  location  has  not  yet  been  determined,  but  its  function  will  be  to  support  the  wide 
area  network  to  conduct  simultaneous  distributed  and  stand-alone  mission  events.  For 
example,  the  four-ship  of  Langley  F-15C  simulators  can  link  with  a  two-ship  of  Eglin  F- 
15s,  controlled  by  a  WD  at  Tinker — all  working  together  in  the  same  “synthetic 
battlespace.”  Meanwhile,  the  remaining  two  simulators  at  Eglin  can  conduct  a  mission 
with  another  WD  at  Tinker,  or  they  can  each  conduct  their  own  stand-alone  missions.1 

ACC  will  continue  expanding  DMT  to  include  MTCs  for  every  combat  unit  weapon 
system.  An  F-16  Block  50  MTC  is  already  scheduled  for  completion  in  2001.  ACC  has 
prioritized  the  rest  of  its  weapons  systems  as  either  high,  medium,  or  low  in  priority, 
based  on  the  benefit  crewmembers  would  gain  from  exchanging  information  with  other 
weapon  systems  in  a  mission  environment — there  is  no  desire  to  add  systems  simply  to 
act  as  training  aids  for  others.  High  priority  weapon  systems  include  the  A-10,  F-15E,  F- 
16  Block  40,  E-8,  F-22,  JSF,  and  EC-130H  ABCCC  and  RC-135  Rivet  Joint  mission 
crews.  Medium  priority  was  assigned  to  F-16  Block  30,  F-117,  B-l,  B-2,  B-52,  and  EC- 
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130E  Compass  Call  mission  crews.  Within  these  categories,  priority  will  go  first  to  units 
most  in  need  of  upgraded  simulators.'  By  2006,  all  high  and  medium  priority  systems  in 
the  CAF  (including  forces  in  Europe  and  the  Pacific)  should  be  incorporated  into  DMT. 

The  combined  DMT  network  will  have  the  capability  to  provide  crew  training  at  all 
levels,  to  include  basic  and  high-end  individual  skills,  team  skills  within  a  unit,  and  inter¬ 
team  skills  among  multiple  MTCs  from  all  over  the  world.4  The  team  building  concept  is 
particularly  important  for  ACC,  which  is  transitioning  to  organized  Air  Expeditionary 
Forces  (AEF).  AEFs  combine  squadrons  from  bases  all  over  the  world  into  pre-planned 
packages  to  deploy  together  if  called  upon  for  military  contingencies.  DMT  will  allow 
them  to  train  together  without  having  to  physically  collect  themselves  into  a  single 
location  for  exercises.  The  AEF  plan  is  still  being  refined,  but  is  currently  planned  for  10 
AEF  teams — two  AEFs  will  be  on  call  at  any  given  time.5  DMT  will  be  able  to  replicate 
the  terrain,  weather,  imagery,  and  opposition  forces  expected  in  the  real-world  theaters  of 
operation,  making  it  an  ideal  training  aid  for  AEF  pre-deployment  preparation.  Mission 
rehearsals  can  prepare  units  to  work  together  before  they  ever  have  to  encounter  a  real 
threat;  they  can  also  benefit  commanders  in  developing  options  to  counter  various  threat 
responses.6 

Expansion  of  DMT  will  go  even  beyond  the  synthetic  battlespace.  Flight  parameters 
and  positions  of  real  fighter  aircraft  can  be  tracked  and  relayed  into  the  network  by 
attaching  specialized  tracking  pods  to  those  aircraft.  This  capability  has  already  been 
demonstrated,  and  the  AW  ACS  MTC  has  in  fact  already  been  used  to  control  live-fly 
missions  in  airspace  near  Eglin  AFB.7  Although  pilots  in  flight  simulators  may  not 
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coordinate  directly  with  live-fly  missions,  command  and  control  centers  can  combine 

o 

both  synthetic  and  live-fly  missions  for  large-scale  battlestaff  training  exercises. 


Notes 

1  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  Distributed  Mission  Training,”  13  October  1998,  11-13. 

~  Air  Combat  Command,  “Roadmap  for  Distributed  Mission  Training  (DMT),” 
Strawman,  6  July  1998 

a 

Gen  Richard  E.  Hawley,  commander,  Air  Combat  Command,  keynote  address  at 
the  Interservice/Industry  Training,  Simulation  and  Education  Conference,  Orlando,  Fla., 
1  December  1998. 

4  HQ  ACC/DOTO,  “Concept  of  Operations  for  Distributed  Mission  Training,”  5-6. 

5  Hawley. 

6  HQ  ACC/DOTO,  “Concept  of  Operations  for  Distributed  Mission  Training,”  7-8. 

7  Lt  Col  J.  A.  Bell,  33  OSS  Current  Operations  Flight  Commander,  previously  HQ 
ACC/DOTO  F-15  Functional  Manager,  telephone  conversation  with  author,  2  October 
1998. 

8  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  F-15C  Mission  Training  Centers,”  Working  Draft,  29 
September  1998,  21. 
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Appendix  C 


Future  Training  Effectiveness  Studies 

Difficulty  in  Conducting  TOT  Studies 

While  TOT  studies  are  theoretically  the  only  sufficient  test  of  simulator 
effectiveness,  they  are  extremely  difficult  to  coordinate  and  conduct.  Simulator  training 
is  believed  to  be  most  effective  for  tasks  difficult  or  impossible  to  practice  in  peace-time 
CT;  attempting  to  demonstrate  transfer  for  more  common  tasks  tend  to  lead  to  only  small 
effects,  as  seen  in  earlier  TOT  studies.  Setting  up  in-flight  tests  for  comparing 
experimental  and  control  groups  in  operational  units  can  also  be  extremely  complex  and 
burdensome.  “In  fact,  one  can  argue  that  it  is  virtually  impossible  to  conduct  a  well- 
controlled  transfer  test  within  an  operational  military  environment.”1 

With  the  overwhelming  response  from  survey  and  ISL  studies  that  air  combat 
simulation  is  valuable,  there  seems  to  be  little  reason  to  pursue  further  proof — especially 
considering  the  required  effort  and  expected  small  effects.  “Given  the  previous  transfer 
of  training  research  that  has  already  been  conducted  ...  there  is  little  reason  to  suspect 
that  such  training  within  a  multiship  simulation  environment  would  not  have  a  positive 
effect  upon  subsequent  performance  in  the  air.”  (Emphasis  in  original)'  A  long-term 
study  is  more  likely  to  provide  measurable  effects  to  compare  DMT  with  comparable 
flight  training.  Results  from  a  long-term  study  would  provide  valuable  and  reliable  data 
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to  prioritize  training  tasks  and  the  medium  to  use  for  various  tasks,  to  determine  the  best 
combination  of  DMT  and  flight  training. 

Although  it  may  be  tedious,  a  long-term  study  tracking  individual  pilots  or  teams 
(flights,  squadrons,  wings,  or  AEFs)  would  be  useful  for  identifying  training  strengths, 
deficiencies,  or  inefficiencies.  Data  could  be  collected  during  daily  DMT  and  CT 
debriefings.  At  a  minimum,  data  should  be  derived  using  objective  measuring  tools 
(discussed  below)  to  track  mission  success  or  training  effectiveness.  In  addition,  data 
should  include  contextual  information,  such  as:  where  pilots  are  in  their  career;  what  unit 
they’re  in;  what  training  is  available;  what  media  is  used  for  training  and  in  what 
combination;  and  what  constraints  have  been  placed  on  flight  training.  With  patience,  a 
long-term  study  would  provide  a  larger  sample  group  for  more  reliable  results,  and 
provide  a  framework  for  better  understanding  how  to  achieve  maximum  training  results 
with  the  available  means. 

F-15C  MTC  Training  Effectiveness  Study 

Air  Force  Research  Faboratory  (AFRF)  is  tasked  by  the  commander  of  Air  Combat 
Command  (COMACC)  to  investigate  DMT  effectiveness,  transfer  of  RAP  training 
elements,  and  “to  determine  the  amount  of  credit  that  may  be  given  for  experience  in  the 
DMT  environment.”  The  training  effectiveness  and  RAP  transfer  investigations  will  be 
the  combination  of  an  in-simulator  learning  (ISF)  study  and  a  transfer  of  training  (TOT) 
study,  measuring  pilot  performance  during  all  MTC  and  aircraft  training,  and  comparing 
both  MTC  pre/post  tests  and  aircraft  pre/post  tests.  This  information  will  be  used  to 
“determine  which  skills  are  better  trained  in  the  MTC  . . .  which  skills  are  better  trained  in 
the  aircraft,  and  ...  determine  the  transferability  of  these  skills  from  the  MTC  to  the 
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aircraft.”4  The  study  will  also  provide  input  to  the  ACC/DO  for  the  MTC  accreditation 
decision  to  determine  which  RAP  requirements  can  be  accomplished  in  the  MTC.5 

For  this  upcoming  study,  AFRL  is  developing  scenarios  for  MTC  training  designed 
to  focus  on  “mission  essential  elements  that  should  receive  the  highest  level  of  attention”6 
due  to  their  difficulty.  Subjective  and  objective  inputs  were  collected  to  determine  which 
elements  were  most  difficult  to  train.  Subjective  data  came  from  subject  matter  expert 
(SME)  inputs.  Objective  data  came  from  ACC  Form  206,  flight  lead  upgrade  (FLUG) 
gradesheets,  collected  during  normal  FLUG  training  at  Eglin  AFB.  Unfortunately,  there 
was  difficulty  identifying  training  shortfalls  from  the  forms  due  to  the  tendency  of 

H 

instructors  to  grade  events  as  “average.”  (This  problem  was  also  mentioned  in  chapter 
three.) 

Regardless  of  how  training  shortfalls  are  identified,  the  concept  of  developing  DMT 
scenarios  to  focus  on  identified  training  shortfalls  should  serve  as  a  model  for  units 
developing  MTC  training  profiles  to  meet  their  specific  training  needs.  As  a  simple 
example,  if  sorting  is  identified  as  a  shortfall,  more  heavy  groups  (with  three  or  more 
threat  aircraft)  should  be  incorporated  into  training  profile  scenarios.  If  a  more  refined 
means  can  be  developed  for  measuring  performance  and  identifying  training  shortfalls, 
training  profiles  can  be  better  designed  to  meet  unit  requirements. 

DMT  Long-Term  Study 

After  AFRL  conducts  the  MTC  Training  Effectiveness  study,  they  will  continue  with 
a  long-term  DMT  study.  Although  this  study  is  “not  yet  specifically  defined,”  it  will 
“compare  performance  of  pilots  with  similar  experience  levels  in  the  aircraft,  but 
different  experience  levels  in  the  DMT  environment  through  naturally  occurring  events 
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such  as  Red  Flag,  Green  Flag,  and  Weapons  School  attendance.”8  Ultimately  the 
performance  comparisons  will  help  determine  how  much  experience  credit  can  be 
awarded  to  pilots  for  missions  flown  in  DMT,  and  may  result  in  a  reallocation  of  flying 
and  DMT  hours.  “Using  information  from  all  three  areas  of  investigation,  ACC/DOTO 
may  be  able  to  recommend  a  re-allocation  of  flying  hours,  in  conjunction  with  simulation 
hours,  that  does  not  decrease  aircrew  mission  readiness  or  aircrew  safety.”9 

Finding  Objective  Measures 

One  of  the  major  problems  for  future  multiship  simulator  research  is  finding 
objective  criteria  to  measure  performance.10  At  this  stage  in  research  development, 
researchers  are  looking  for  ways  not  just  to  declare  that  performance  has  improved,  but 
how  and  why  has  it  improved.  AFRL  went  through  years  of  preparation  and  several 
independent  studies  to  develop  the  behavioral  indicators  for  SA  Rating  Scales  (SARS)  in 
their  most  recent  ISL  study.11  The  resulting  behavioral  indicators  look  remarkably  like 
current  F-15C  gradesheets  items  used  for  upgrade  training.  They  also  resemble  a  briefing 
or  debriefing  outline  used  by  flight  leads  and  IPs  on  everyday  CT  flights.  This  similarity 
adds  confidence  that  the  processes  listed  are  on  the  right  track  for  measuring  the 
component  tasks  required  to  achieve  mission  success. 

For  their  upcoming  F-15C  MTC  training  effectiveness  study,  AFRL  developed  an 
even  longer  list  of  tasks  to  be  graded,  incorporating  briefing,  pre-push,  ingress,  egress, 
and  debriefing  elements.  Although  the  list  may  be  cumbersome  to  complete,  these 
measurement  criteria  should  provide  valuable  data  for  performance  comparisons. 
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Soft  Measures  for  Team  Success 


What  behavioral  indicators  and  the  new  list  still  fail  to  identify  are  the  more  subtle 
processes  that  must  be  learned  and  exercised  to  ensure  team  success — which  is  of  utmost 
importance  for  DMT.  Developed  measures  may  still  fall  short  of  measuring  true 
effectiveness.  The  high-payoff  items  in  advanced  simulators  are  things  that  make  up 
expertise ,  which  are  hard  to  measure.  Expertise  involves  team  and  collective  behaviors 
(such  as  leadership  or  trust  in  the  flight  lead),  and  decision-making  (such  as  gameplan 
and  tactics  selection,  or  changing  the  gameplan  when  things  fall  apart).  Future  tests  may 

be  satisfied  with  specifying  sub-level  motor  skills  and  miss  the  more  important  “soft 

1 2 

measures,”  where  the  real  impacts  are. 


Notes 

1  Wayne  L.  Waag  and  Herbert  H.  Bell,  Estimating  the  Training  Effectiveness  of 
Interactive  Air  Combat  Simulation,  report  no.  AL/HR-TP- 1996-039  (Mesa,  Ariz.: 
Armstrong  Laboratory,  Human  Resources  Directorate,  February  1977),  6. 

2  Ibid.,  7. 

"5 

Ronald  D.  Dunlap,  “The  Training  Effectiveness  Evaluation:  Measuring  the 
Effectiveness  of  Training  in  an  F-15C  Mission  Training  Center,”  Air  Force  Research 
Laboratory,  Warfighter  Training  Research  Division,  n.d.,  3. 

4  Ibid.,  5. 

5  Air  Combat  Command,  Operations  and  Training  Branch  (HQ  ACC/DOTO), 
“Concept  of  Operations  for  F-15C  Mission  Training  Centers,”  Working  Draft,  29 
September  1998,  22. 

6  Dunlap,  6. 

7 

Dr.  Ronald  D.  Dunlap,  telephone  conversation  with  author,  26  February  1999. 

8  Dunlap,  3-4. 

9  Ibid.,  3. 

10  Michael  R.  Houck,  Leslie  A.  Whitaker,  and  Robert  R.  Kendall,  An  Information 
Processing  Classification  of  Beyoncl-Visual-Range  Air  Intercepts,  report  no.  AL/HR  TR- 
1993-0061/AD  A266927  (Williams  Air  Force  Base,  Ariz.:  Armstrong  Laboratory, 
Aircrew  Training  Research  Division,  May  1993). 

1 1  Wayne  L.  Waag  et  al.,  “Use  of  Multiship  Simulation  as  a  Tool  for  Measuring  and 
Training  Situation  Awareness,”  in  AGARD-CP-575,  Situation  Awareness:  Limitations 
and  Enhancement  in  the  Aviation  Environment  (Neuilly-Sur-Seine,  France:  Advisory 
Group  for  Aerospace  Research  and  Development,  January  1996). 
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‘  Dr.  Herbert  H.  Bell,  telephone  conversation  with  author,  19  November  1998. 
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